Dense field imaging

ABSTRACT

Dense field imagers are disclosed which are configured to provide fused and/or stitched light field data for a scene. A dense field imager can include a plurality of imaging elements configured to be joined into image blocks or facets that each provides light field data about a scene. The dense field imager can include a plurality of facets in a fixed or modular fashion such that the dense field imager is configured to fuse and/or stitch light field data from the plurality of facets. The facets can be mounted such that one or more facets are non-coplanar with other facets. The facets can be configured to provide light field data with overlapping fields of view. Accordingly, the dense field imager can provide dense field data over a field of view covered by the plurality of facets.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/181,552, filed Feb. 14, 2014, and titled “DENSE FIELD IMAGING,” whichapplication claims the benefit of priority to U.S. Provisional PatentApplication No. 61/765,661, entitled “DENSE FIELD IMAGING,” filed Feb.15, 2013, and to U.S. Provisional Patent Application No. 61/785,494,entitled “IMAGING SYSTEM WITH MULTIPLE IMAGING DEVICES HAVINGOVERLAPPING FIELDS OF VIEW,” filed Mar. 14, 2013. Each applicationreferenced in this paragraph is incorporated by reference herein in itsentirety so as to form part of this specification.

BACKGROUND

The present disclosure generally relates to acquiring and processingimage data from a plurality of imaging devices, and particularly toacquiring and combining light field representations to form a densefield data set, and generating images from the dense field data set.

SUMMARY

The systems, methods and devices of the disclosure each have innovativeaspects, no single one of which is indispensable or solely responsiblefor the desirable attributes disclosed herein. Without limiting thescope of the claims, some of the advantageous features will now besummarized.

Dense field imagers are disclosed which are configured to provide aplurality of light field representations of a scene. Dense field imagerscan be configured to combine the plurality of light fieldrepresentations to generate a dense field data set. A dense field imagercan include a plurality of imaging elements configured to be joined intoimage blocks or facets, the imaging blocks providing data that can beused to generate light field representations for a scene. The densefield imager can include a plurality of image blocks or facets in afixed or modular fashion such that the dense field imager is configuredto combine light field representations from the plurality of facets. Thefacets can be mounted such that one or more facets are non-coplanar withother facets. The facets can include processing modules that areconfigured to generate light field representations from acquired pixeldata. The generated light field representations can have overlappingfields of view. Accordingly, the dense field imager can provide densefield data over a field of view covered by the plurality of facets.

Image processing systems are also disclosed which are configured toreceive light field representations from a plurality of sources and tocombine the light field representations to generate a unitary, densefield data set. The generated dense field data set can be used todynamically generate viewpoints or images having a variety of desiredproperties. For example, a depth of focus can be changed dynamically,stereoscopic images can be provided where an inter-ocular distance canbe changed dynamically, images from a variety of virtual viewpoints canbe generated, and the like. These image processing systems can allowdynamic editing and cinematography such that the importance of theconfiguration of the cameras acquiring the data is reduced as many ofthe creative decisions can be made during post-processing. For example,using the image processing systems described herein a director or videoeditor could re-light a scene virtually, perform depth-based editing orcoloring, shift a focus of the image, zoom in or out of a scene, changea viewpoint of the scene, and the like.

In some embodiments, the image processing system can allow for users topersonalize videos or movies. For example, a movie can be made usingdata acquired with the dense field imager described herein. The densefield image data can then be provided to allow a person not associatedwith the original creation of the movie to change characteristics of themovie by changing focus, viewpoint, lighting, zoom, etc.

In some embodiments, the dense field imager and/or image processingsystems described herein can allow for a reduction or elimination of theinvolvement of camera operators in a venue. For example, one or moredense field imagers can be provided in a stadium at an athletic eventand a director can dynamically change a focus of output images to focuson an object or region of interest. In some embodiments, a dense fieldimager can be configured to dynamically track or focus on one or moreobjects or persons of interest. It can be possible, for example, then torefocus on different objects according to focus criteria.

Some embodiments described herein provide for a dense field imagingblock and array. The imaging block can include a support and at least afirst imaging element and a second imaging element carried by thesupport, each imaging element comprising a sensor and a lens. Theimaging block can include a mechanical connector for mechanicallyconnecting the imaging block into an array of imaging blocks. Theimaging block can include an electrical connector for electricallyconnecting the imaging block into an array of imaging blocks.

In some implementations, the first imaging element of the imaging blockcomprises a monochromatic filter, wherein the sensor of the firstimaging element detects substantially monochromatic light passingthrough the monochromatic filter. In some implementations, the supportcomprises a wafer substrate and the wafer substrate can sometimes bemade of a semiconductor material. In some implementations, the sensorsare formed on the substrate. In some implementations, the lenses arewafer-level lenses.

Some embodiments provide for an imaging array that includes an arraysupport and at least two imaging blocks carried by the array support.The at least two imaging blocks can include a first imaging block and asecond imaging block. At least one of the sensors in the first imagingblock can be non-coplanar with at least one of the sensors in the secondimaging block. In some implementations, each of the sensors in the firstimaging block are coplanar, each of the sensors in the second imagingblock are coplanar, and each of the sensors in the first imaging blockare non-coplanar with each of the sensors in the second imaging block.In some implementations, each imaging element has a primary opticalaxis, and the primary optical axis of at least one of the imagingelements in the first imaging block is substantially non parallel withthe primary optical axis of at least one of the imaging elements in thesecond imaging block. In some implementations, each imaging element hasa primary optical axis, the primary optical axes of the imaging elementsin the first imaging block are substantially parallel, the primaryoptical axes of the imaging elements in the second imaging block aresubstantially parallel, and the primary optical axes of the imagingelements in the first imaging block are substantially non-parallel withthe primary optical axes of the imaging elements in the second imagingblock. In some implementations, a primary optical axis of at least oneimaging element of the first imaging block is angularly adjustable withrespect to a primary optical axis of at least one imaging element of thesecond imaging block. In some implementations, the imaging arraycomprises a user-actuatable control for achieving the angularadjustment.

The imaging array can also include, in some implementations, an imageprocessing system configured to use image data captured by the firstimaging block to generate a first image data set representative of afirst portion of a light field. It can also be configured to use imagedata captured by the second imaging block to generate a second imagedata set representative of a second portion of the light field. It canalso be configured to derive a third image data set from the first andsecond image data sets. In some implementations, the image processingsystem is carried by the array support. In some implementations, theimage processing system is physically separate from the array supportand receives the first and second image data sets wirelessly. In someimplementations, the image processing system derives the third imagedata set at least partly by creating a spatial relationship tensor thatincludes spatial relationship information between elements of the firstand second image data sets and using the spatial relationship tensor toderive the third image data set. In some implementations, the imageprocessing system derives the third image data set at least partly byusing the spatial relationship tensor to combine together the first andsecond portions of the light field.

In some implementations, the imaging block to include one or moreprocessors carried by the support and configured to generate a lightfield representation based on pixel data acquired from the sensors. Insome implementations, one or more of the lenses are removablyreplaceable with lenses having different optical characteristics. Insome implementations, each imaging element has a primary optical axis,and most of the primary optical axes are substantially parallel. In someimplementations, each imaging element has a primary optical axis, and atleast two of the primary optical axes diverge in a direction leadingaway from the sensor. In some implementations, each imaging element hasa primary optical axis, and at least two of the primary optical axesconverge in a direction leading away from the sensor. In someimplementations, each imaging element has a primary optical axis, and atleast a first primary optical axis is angularly adjustable with respectto at least a second primary optical axis. In some implementations, theimaging block includes at least 8 imaging elements. In someimplementations, the imaging elements are arranged in two rows of 4. Insome implementations, the imaging block includes at least 16 imagingelements. In some implementations, the imaging elements are arranged ina 4×4 grid. In some implementations, at least one sensor is no largerthan about 5 mm×5 mm. In some implementations, the imaging block alsoincludes an FPGA chip carried by the support. In some implementations,at least some of the sensors have one or more of different sizes,different resolutions, or different sensitivities.

Some embodiments provide for an imaging array comprising an arraysupport, and at least two imaging blocks according to any of the abovedescriptions, the imaging blocks being carried by the array support.

Some embodiments provide for a method of compiling an image data set.The method can include obtaining a first image data set representativeof a first portion of a light field. The method can include obtaining asecond image data set representative of a second portion of the lightfield. The method can include deriving, with one or more processors, athird image data set that is based on at least the first image data set,the second image data set, and information relating to a spatialrelationship between the first and second portions of the light field.The method can include storing the third image data set in one or morememory devices.

In some implementations, the first image data set is derived from pixeldata acquired by a first group of at least two imaging elements and thesecond image data set is derived from pixel data acquired by a secondgroup of at least two imaging elements, each imaging element comprisinga sensor and a lens. In some implementations, the method also includesprocessing the first image data set and the second image data set todetermine the spatial relationship between the first portion of thelight field and the second portion of the light field. In someimplementations, deriving includes accessing a spatial relationshiptensor representing the spatial relationship information to perform ageometric transform on the first image data set and second image dataset. In some implementations, the at least two imaging elements in thefirst group are coplanar, the at least two imaging elements in thesecond group are coplanar, and the at least two imaging elements in thefirst group are non-coplanar with respect to the at least two imagingelements in the second group. In some implementations, the first andsecond image data sets additionally represent the first portion and thesecond portion of the light field as a function of time. In someimplementations, the third image data set comprises light fieldinformation represented as a function of time. In some implementations,the first portion of the light field and the second portion of the lightfield comprise regions of the light field which at least partiallyoverlap, and wherein the third data set comprises light fieldinformation derived from data in both of the first and second data setsthat corresponds to the region of the light field lying within theoverlap. In some implementations, the first portion of the light fieldand the second portion of the light field comprise regions of the lightfield which only partially overlap. The third data set can include lightfield information derived from data in the first image data set thatcorresponds to a first portion of a scene, and data in the second imagedata set that corresponds to a second portion of the scene that does notoverlap with the first portion.

In some implementations, the third image data set comprises at least 4Dlight field information. In some implementations, deriving the thirdimage data set comprises deriving the third image data set whilemaintaining the dimensionality of functions that represent the first andsecond portions of the light field. In some implementations, the firstand second image data sets respectively represent the first and secondportions of the light field as functions having at least four inputparameters. In some implementations, the first and second image datasets respectively represent the first and second portions of the lightfield as functions having at least five input parameters. In someimplementations, the first and second image data sets respectivelyrepresent the first and second portions of the light field as functionsthat represent luminance as a function of a position in space and apointing direction.

In some implementations, viewable images are extractable from the thirdimage data set. In some implementations, the viewable images comprise 2Dimages and/or 3D images. In some implementations, viewable motion videois extractable from the third image data set.

In some implementations, the method can also include accessing a fourthimage data set representative of a third portion of the light field,wherein said deriving comprises deriving the third image data set basedon at least the first image data set, the second image data set, thefourth image data set, and spatial relationships between the first,second, and third portions of the light field.

Some embodiments provide for a stored image data set on a memory devicewherein the memory device includes a dense field image data set. Thememory device can include a storage medium a dense field image data setstored in the storage medium and derived by relating a first image dataset representative of a first portion of a light field to a second imagedata set representative of a second portion of the light field, usinginformation relating to a spatial relationship between the first andsecond light fields.

Some embodiments provide for a computer-readable memory device, whereinthe image data set comprises light field information represented as afunction of time. In some implementations, the computer-readable memorydevice includes the image data set and the image data set includes atleast 4D light field information. In some implementations, the firstimage data set was derived from pixel data acquired by a first imagingblock comprising a support and at least two imaging elements carried bythe support, and the second image data set was derived from pixel dataacquired by a second imaging block comprising a support and at least twoimaging elements carried by the support, each imaging element comprisinga sensor and a lens.

Some embodiments provide for a machine comprising the computer-readablememory device above and one or more processors configured to derive thedense field image data set. In some implementations, the one or moreprocessors configured to extract viewable images from the dense fieldimage data set.

Some embodiments provide for a computer-readable memory device havingviewable images that are 2D images, 3D images, and/or motion video.

Some embodiments provide for a method of creating a dense field imageset. The method can include acquiring pixel data from a plurality ofimaging elements, each imaging element comprising a sensor and a lens.The method can include generating a plurality of light fieldrepresentations, each of the light field representations generated usingpixel data from at least two of the imaging elements. The method caninclude creating a spatial relationship tensor representative of spatialrelationships among the light field representations. The method caninclude utilizing the spatial relationship tensor, combining the lightfield representations to create a dense field image set.

In some implementations, combining the light field representationscomprises stitching together light field representations having at leastpartially non overlapping fields of view to create a dense field imageset having a significantly wider field of view than the individual lightfield representations. In some implementations, the lateral field ofview of the dense field image set is greater than or equal to about 145degrees. In some implementations, the imaging elements are arranged on acommon support. In some implementations, at least some of the imagingelements are not coplanar. In some implementations, each light fieldrepresentation is generated using pixel data from an imaging blockcomprising at least two imaging elements. In some implementations, atleast some of the imaging blocks are not coplanar. In someimplementations, at least one of the imaging elements comprises amonochromatic filter, and wherein light passing through themonochromatic filter is detected by the sensor of the at least oneimaging element. In some implementations, at least half of the imagingelements comprise monochromatic filters. In some implementations,substantially all of the imaging elements comprise monochromaticfilters.

Some embodiments provide for an imaging system configured to generate alight field representation for each of a plurality of imaging blocks.The plurality of imaging blocks each can include at least two imagingelements, and each of the imaging elements can include an image sensorand a lens. The imaging system can include a dense field image processormodule configured to, for each imaging block of at least two of theplurality of imaging blocks, generate pixel correspondence informationfor the imaging block, the pixel correspondence informationrepresentative of spatial relationships between pixels in each imagingelement of the imaging block and corresponding pixels in other imagingelements of the imaging block. The module can be configured to utilizethe correspondence information to generate a light field representationusing pixel data acquired by the respective imaging block.

In some implementations, the dense field image processor module isfurther configured to create a spatial relationship tensorrepresentative of spatial relationships among the light fieldrepresentations, and, utilizing the spatial relationship tensor, combinethe light field representations to create a dense field image set.

In some implementations, the imaging blocks are arranged on a commonsupport. In some implementations, the imaging blocks are formed in awafer. In some implementations, the imaging elements of each imagingblock are coplanar with respect to one another. In some implementations,at least some of the imaging blocks are not coplanar with respect toother ones of the imaging blocks. In some implementations, at least someof the imaging blocks are arranged on physically separate supports.

Some embodiments provide for a dense field imaging system that includesa plurality of imaging blocks, each of the imaging blocks comprising atleast two sensor/lens pairs, wherein at least some of the imaging blocksare substantially non-coplanar with each other and have at leastpartially non-overlapping fields of view. The dense field imaging systemcan include a dense field image processor module configured to, for eachimaging block, generate a light field representation using pixel dataacquired by the sensor/lens pairs. The module can be configured togenerate a spatial relationship tensor representative of spatialrelationships between the light field representations. The module can beconfigured to use the spatial relationship tensor to combine the lightfield representations to create a dense field imaging set having asubstantially wider field of view than the individual light fieldrepresentations. In some implementations, each of the imaging blockscomprises a wafer substrate on which the sensors are formed. In someimplementations, the dense field imager also includes a controlconfigured to adjust the angular relationship between sensor/lens pairs.

Some embodiments provide for a dense field imaging system that includesa plurality of imaging blocks arranged on a support, each of the imagingblocks comprising at least two imaging elements, each of the imagingelements comprising a sensor and lens, wherein at least some of theimaging elements comprise monochromatic filters. The dense field imagingsystem can include a dense field image processor module configured to,for each imaging block, generate a light field representation usingpixel data acquired by the imaging elements of the imaging block. Themodule can also be configured to combine the light field representationsto create a dense field image set.

In some implementations, most of the sensors are monochromatic. In someimplementations, each of the sensors is monochromatic. In someimplementations, each of the imaging blocks includes a supportcomprising a wafer substrate, and wherein the sensors are formed on thewafer substrate. In some implementations, the lenses comprise waferlevel lenses.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are provided to illustrate example embodiments describedherein and are not intended to limit the scope of the disclosure.Throughout the drawings, reference numbers may be re-used to indicategeneral correspondence between referenced elements.

FIG. 1A illustrates a structural block diagram of an embodiment of animaging system having a plurality of imaging blocks or facets, eachfacet having a plurality of imaging elements.

FIG. 1B illustrates a functional block diagram of an embodiment of animaging system, such as a dense field imager, coupled to a dense fieldprocessing system.

FIG. 2 illustrates an example imaging element that includes a lens, animage sensor, and image acquisition electronics.

FIG. 3 illustrates an example of a single facet of a dense field imager,the single facet comprising a plurality of imaging elements withoverlapping fields of view.

FIG. 4 illustrates an example of a facet having a plurality of imagingelements coupled together on a common support, the common supportincluding facet processing electronics.

FIGS. 5A and 5B illustrate schematic drawings showing a dense fieldimager with an array of imaging elements coupled to a common support andacquisition electronics.

FIG. 6A illustrates a field of view for an individual imaging element,such as an imaging element in a dense field imager.

FIGS. 6B and 6C illustrate overlapping fields of view for example arraysof imaging elements that are arranged in rectangular and hexagonalgrids, respectively.

FIG. 7A illustrates imaging elements with partially overlapping fieldsof view.

FIG. 7B illustrates imaging elements with substantially overlappingfields of view.

FIG. 7C illustrates imaging elements with substantially adjacent fieldsof view.

FIG. 7D illustrates imaging elements with partially overlapping fieldsof view, where the angle of view is decreased.

FIG. 7E illustrates imaging elements with substantially overlappingfields of view, where the angle of view is decreased.

FIG. 7F illustrates imaging elements with substantially adjacent fieldsof view, where the angle of view is decreased.

FIG. 8 illustrates an example dense field imager having an array ofimaging elements wherein each imaging element includes a monochromaticsensor.

FIGS. 9A-C illustrate an example embodiment of a planar dense fieldimager having integrated cooling elements and integrated acquisition andprocessing electronics.

FIGS. 10A-C illustrate an example embodiment of a dense field imagerhaving facets that are mounted in a concave fashion.

FIGS. 10D-F illustrate fields of view of some of the facets on the densefield imager illustrated in FIGS. 10A-C.

FIGS. 11A-C illustrate an example embodiment of a dense field imagerhaving facets that are mounted in a convex fashion.

FIGS. 11D-F illustrate fields of view of some facets on the dense fieldimager illustrated in FIGS. 11A-C.

FIGS. 12A and 12B illustrate perspective views of an example dense fieldimager including a plurality of imaging elements arranged in an array.

FIG. 12C shows a side perspective view of an embodiment of a dense fieldimager having a plurality of imaging elements with large, removablelenses.

FIG. 12D shows a side perspective view of an embodiment of a dense fieldimager having a plurality of imaging elements with small, removablelenses.

FIG. 12E shows a side perspective view of an embodiment of a dense fieldimager having a plurality of imaging elements with non-removable,low-profile lenses.

FIGS. 12F-H show top perspective views of a dense field imager with agrip attachment, the dense field imager having a plurality of imagingelements with, respectively, large, removable lenses; small, removablelenses; and non-removable, low-profile lenses.

FIGS. 12I and 12J show top perspective views of a dense field imager,respectively with and without a grip attachment, showing the viewfinderscreen.

FIG. 13 illustrates an embodiment of a dense field imager that isconfigured to wirelessly transmit acquired data to an image hub.

FIGS. 14A and 14B respectively illustrate a top plan view and a sideelevational view of a multi-imaging device system.

FIG. 14C illustrates a top plan view of an array of cameras recording anobject or other portion of interest in a viewing theater.

FIG. 15 illustrates a block diagram of an example image processingsystem for receiving data from a plurality of sources of different typesand generating light field data, dense field data, and/or viewpointsfrom received data.

FIG. 16 illustrates a flow chart of an example method for acquiringimage data with a dense field imager or an array of imaging deviceshaving overlapping fields of view wherein a plurality of light fieldrepresentations are generated using two or more imaging elements orimaging devices.

FIG. 17 illustrates a flow chart of an example calibration method fordetermining corrective factors for imaging elements on a dense fieldimager or an array of imaging devices.

FIG. 18A illustrates an example object that can be used to in thecalibration method illustrated in FIG. 17.

FIG. 18B illustrates an example plot showing results of corrected andnon-corrected pixel data.

FIG. 18C illustrates an example plot of relative imaging elementorientation.

FIG. 18D illustrates a plot of a result of calibrating imaging elementson a dense field imager.

FIG. 19 illustrates a flow chart of an example method 1900 for stitchinglight field representations.

FIGS. 20A and 20B represent a scene to be captured by a plurality ofdevices configured to generate light field representations and a fieldof view of one of the plurality of devices.

FIG. 21 illustrates a timing diagram for capturing images using multipleimaging devices.

FIG. 22 illustrates a flow chart of an example method for generating aviewpoint using dense field data.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings. It is to be understood that other structures and/orembodiments may be utilized. Various aspects of the disclosure will bedescribed with regard to certain examples and embodiments, which areintended to illustrate but not to limit the disclosure. Nothing in thisdisclosure is intended to imply that any particular feature orcharacteristic of the disclosed embodiments is essential.

Generally, image sensors that capture digital images are made up ofmultiple picture elements, or pixels. Each pixel receives light that isdirected to it by an optical system. The light received by the pixelcreates an electrical charge, and the cumulative charge caused by lightreceived by the pixel is converted into a digital value. By combininginformation from multiple image sensors, multiple capabilities can beprovided that, for example, improve image quality and/or provideinformation that generally is not obtained using a single image sensor.For example, dynamic range can be increased, noise can be reduced,directional and/or depth information can be extracted, focus can beadjusted, resolution can be enhanced, and the like.

A plurality of imaging elements having their own optics and imagesensors can be combined to provide advantages outlined above anddescribed in greater detail herein. A plurality of imaging elements canbe combined into an image block where each imaging element providesimage data from their sensors. The image data from the plurality ofimaging elements in the image block can be combined to produce a lightfield representation over a field of view. Multiple image blocks can becombined or grouped to form an imaging system, such as a dense fieldimager, which is configured to combine light field representations froma plurality of image blocks.

Image processing systems can also be configured to receive image datafrom a plurality of imaging elements to produce light fieldrepresentations. These or other image processing systems can also beconfigured to receive light field representations covering multiplefields of view and combine the light field representations to enhance orincrease information contained in the combined light fieldrepresentation and/or increase a field of view of the combined lightfield representation.

As will be described in further detail, imaging systems described hereincan capture pixel data that can be used to generate one or more lightfield representations, and in some embodiments provide similaradvantages to light-field and plenoptic cameras, in addition toproviding other advantages. Such advantages can include the capabilityto adjust focus and/or depth of field in post-processing. For instance,some of the multi-imaging device cameras described herein can capturelight field information about the image scene, which can be used by theby post-processing software (or by the camera system) to provideadjustment of focus, depth of field, and other characteristics. Suchadjustments can be made after recording, such as during playback orediting.

As used herein, the terms light field data, light field representation,and the like can be used to mean, for example and without limitation, arepresentation of luminance or radiance as a function of space,direction, wavelength, and/or time. A light field can be used to meanthe radiance or luminance of light within a scene, independent of anydevice used to capture light-related information. In some embodiments, alight field can be represented using a function with at least fivedimensions as input wherein the dimensions can include three spatialdimensions and two directional dimensions. For example, the spatialdimensions can be coordinates in a Cartesian coordinate system (e.g., x,y, and z) and the directional coordinates can be angles in a sphericalcoordinate system (e.g., θ and φ, or azimuth and elevation). In someembodiments, a light field can be represented using a function with atleast four dimensions as input where the spatial and directionaldimensions are represented using a parameterization such as a lightslab, light spheres, points on a plane with an angle, or the like,reducing the number of dimensions used to represent position anddirection from five to four. A light field representation can include awavelength as input, thereby representing luminance or radiance as afunction of wavelength in addition to position and direction. A lightfield representation can include time as input, thereby representingluminance or radiance as a function of time in addition to position anddirection. A light field representation can include both time andwavelength as input in addition to position and direction. In someembodiments, a separate light field representation can be used fordifferent wavelength bands (e.g., a blue light field representation, agreen light field representation, a red light field representation, abroadband light field representation, etc.). The light fieldrepresentations or light field data, then, can be used to approximate orrepresent a portion of the light field, e.g., the portion of the lightfield within a scene or region imaged by one or more imaging devices.

As used herein, the terms dense field, dense field image data, densefield image set, dense field representations, and the like can be usedto mean, for example and without limitation, a unitary data setresulting from combining a plurality of input light fieldrepresentations wherein the unitary data set can be used to generatemultiple viewpoints and/or images. The unitary data set can includeinformation about relationships between input light fieldrepresentations, relationships between pixels in imaging elements usedto acquire the input light field data, alignment information, geometrictransformations, and the like.

It is to be understood that data such as light field data and/or densefield image data can include metadata that can be used to provideinformation related to the light field data and/or dense field imagedata. Metadata can be used, for example and without limitation, todescribe a resolution of one or more imaging elements, color depth,image sensor properties, focal length of imaging elements, zoom ofimaging elements, acquisition date and time, aperture settings, densefield imager information, calibration information, and other such data.The metadata can also include information that is associated with acamera operator, a setting in which the data is acquired, a position ororientation of an imaging element or dense field imager, and the like.Metadata can be used to store information that can be used to generateviewpoints, to combine image data, to process acquired data, and thelike.

Dense Field Imager

FIG. 1A illustrates a structural block diagram of an embodiment of animaging system 100, such as a dense field imager, having a plurality ofimaging blocks or facets 300, each facet 300 having an array of imagingelements 200. The dense field imager 100 includes an array support 150configured to position and orient the plurality of facets 300. Asdescribed herein, the dense field imager 100 can have a variety ofconfigurations and can include a plurality of facets 300 that arecoplanar, non-coplanar, or a combination of coplanar and non-coplanarfacets 300. The imaging elements 200 can be grouped into facets 300where the imaging elements 200 are mounted on or attached to a facetsupport. The imaging elements 200 on a given facet 300 can besubstantially coplanar with the other imaging elements 200 on the samefacet or the imaging elements 200 can be non-coplanar such that thefacet support is generally concave, convex, or has some other non-planarshape. In some embodiments, the imaging elements 200 on a particularfacet 300 have optical axes that are substantially parallel. In someembodiments, the optical axes of the facets 300 (e.g., a geometricaverage of the optical axes of the imaging elements 200) can besubstantially parallel, can generally diverge, can generally converge,or any combination of these. In some embodiments, the dense field imager100 includes facets 300 that are not coupled to a common support 150,but are loosely physically coupled or physically separate and logicallyconnected through a common wired or wireless connection to one or moreacquisition or processing systems. In some embodiments, the plurality offacets 300 can be grouped into an imaging system.

FIG. 1B illustrates a functional block diagram of an embodiment of animaging system 100, such as a dense field imager, coupled to a densefield processing system 102. The dense field imager 100 includes aplurality of imaging elements 200, wherein the imaging elements 200 aregrouped into image blocks or facets 300. The dense field imager 100 cancomprise a plurality of facets 300 which are electrically coupled andwhich can share at least a portion of image acquisition electronics.

Each of the plurality of imaging elements 200, as described in greaterdetail herein, can include optics and an image sensor such that theimaging element 200 can provide image data of a scene. For example, theimaging element 200 can be a conventional camera configured to provideraw, processed, and/or compressed image data corresponding to lightcollected from a scene. The plurality of imaging elements 200 can beidentical or they can differ from one another in a variety of ways. Forexample, imaging elements 200 can have different focal lengths, dynamicranges, frame rates, photosensitivity, color gamut, fields of view, andthe like. The imaging elements 200 can be configured to provide video inaddition to images, and it should be understood that throughout thisdisclosure where images or image data is discussed video or video datais included. Thus, in some implementations, the imaging system 100 isconfigured to acquire motion and still images at the same time.

Two or more imaging elements 200 can be combined to form an image blockor facet 300. The plurality of imaging elements 200 in a facet 300 areconfigured to provide image data of a scene to facet acquisition modules304, which can be configured to combine the image data for processing.For example, the facet acquisition module 304 can be configured tocombine image data from the imaging elements 200 to provide light fielddata, dense field data, stitched image data, combined image data, or anycombination of these. In addition, facets 300 can be combined in amodular fashion to dynamically change a size of the imaging system 100.

The facet acquisition modules 304 can be configured to send the outputof the module to a dense field acquisition module 118 of the dense fieldprocessing system 102. The dense field processing system 102 can beconfigured to generate light field data from pixel data received fromthe facet acquisition modules 304. In some embodiments, the dense fieldprocessing system 102 is configured to join light field data receivedfrom the facet acquisition modules 304 to generate dense field data. Insome embodiments, the dense field processing system 102 can beconfigured to perform image processing functions as described herein,such as, for example and without limitation, calibration, registration,filtering, alignment, stitching, fusion, and the like. In someembodiments, the facet acquisition modules 304 perform some or all ofthese functions in addition to or instead of the dense field processingsystem 102 performing them.

The dense field processing system 102 can receive the output of thefacet acquisition modules 304 and further process the image data. Thedense field processing system 102 can include data storage 104 forstoring image data, calibration data, metadata, camera data, and thelike. The dense field processing system 102 can include a controller 106configured to control operation of the components of the system 102. Thecontroller 106 can include one or more computer processors, FPGAs,ASICs, DSPs, or the like. The dense field processing system 102 caninclude a dense field processing module 108 configured to performanalysis and processing functions. The processing module 108 can includehardware, firmware, and/or software components. For example, theprocessing module 108 can be configured to perform executableinstructions stored on a computer readable medium. The processing module108 can include hardware components programmed to perform definedfunctions, such as a FPGA configured to fuse or stitch image data.

The dense field processing system 102 can be configured to provide avariety of processing capabilities through the dense field processingmodule 108. For example, the dense field processing system 102 can beconfigured to fuse image data which can include, without limitation,joining image data at common or coinciding locations to increaseinformation available at those locations. The system 102 can also beconfigured to stitch image data by, for example, identifyingcorresponding features, pixels, or light rays in regions of overlappingimage data and joining image data from two or more imaging elements orfacets based at least in part on the image data in the overlappingregion. The dense field processing system 102 can be configured toenhance a resolution of an output image by, for example, combining aplurality of input images of a scene and performing sub-pixelsharpening. The dense field processing system 102 can be configured toprovide substantially clear imagery of a scene that is partiallyoccluded by combining image data from a variety of viewpoints to piecetogether a complete image substantially free of the occlusion. The densefield processing system 102 can be configured to provide 3D imagery witha configurable inter-ocular distance by providing imagery from two,adjustable viewpoints. The dense field processing system 102 can beconfigured to provide imagery from a variety of viewpoints and/orviewing angles wherein the viewpoints can correspond to locations withimaging elements or the viewpoints can be virtual viewpoints which donot correspond to locations with at least one imaging element. The densefield processing system 102 can be configured to change a depth of fieldand/or a focus depth of an output image. The dense field processingsystem 102 can be configured to provide output imagery having a higherdynamic range than any individual camera or imaging element 200 used toacquire the image data. The dense field processing system 102 can beconfigured to receive image data and/or light field data and outputdense field image data.

The imaging system 100 can be configured to acquire image data in avariety of combinations. Image data from the imaging elements 200 can becombined to provide light field data, to enhance resolution of an imagedscene, to provide image data having a relatively high dynamic range, toproduce video with a high frame rate, and the like. Image data acquiredby the imaging elements 200 can be provided to the dense fieldprocessing system 102 in a raw and/or compressed format, thus allowingthe dense field processing system 102 to manipulate and process theimage data in a variety of manners. For example, the imaging system 100can be configured to record any combination of high resolution images,light field data, dense field data, and/or smaller compressed images orvideo (e.g., jpeg, mpeg, TIFF, etc.). Accordingly, image data from theimaging system 100 can be provided for relatively quick editing ordistribution, and/or it can be combined and stored to create a largerdata set for post-processing and analysis.

In some embodiments, the imaging system 100 can be an array-basedimaging system that includes an array of image blocks or facets 300.Each of the facets 300 can include at least two imaging elements 200comprising at least a sensor and lens pair. The dense field acquisitionmodule 118 can be configured to generate a light field representationfor each facet 300 using pixel data acquired by the imaging elements 200of the facet 300. The dense field acquisition module 118 can beconfigured to utilize alignment information that specifies geometricrelationships between the imaging elements 200 to join light fieldrepresentations by additively combining at least overlapping portions ofthe light field representations to create a dense field image set. Thedense field image set, in some embodiments, can be configured to enablegenerated viewpoints having improved resolution, improved correspondencewith the scene, improved perceived quality, and the like as compared tothe individual light field representations.

In some embodiments, the dense field acquisition module 118 isconfigured to create a spatial relationship tensor that includes spatialrelationships among the light field representations. The dense fieldacquisition module 118 can use the spatial relationship tensor to stitchtogether non-overlapping portions of the light field representations.The output of this process can provide a dense field image set that hasa substantially wider field of view than the individual light fieldrepresentations. In some embodiments, the dense field acquisition module118 is configured to obtain the alignment information by processingcalibration images acquired by the imaging elements 200.

Imaging Elements

FIG. 2 illustrates an example imaging element 200 that includes a lens210, image sensor 220, and image acquisition electronics 230. When usedin the imaging system 100, e.g., a dense field imager, the output of theimage acquisition electronics 230 can be sent to a facet acquisitionmodule 304 configured to join image data from a plurality of imagingelements 200. The example imaging element 200 can be a conventionalcamera configured to produce a color or monochromatic image or video ofa scene.

In some embodiments, a dense field imager includes a plurality ofimaging elements 200. The imaging elements 200 can be arranged, forexample, in an array, attached to one or more supports. The supports canposition the imaging elements 200 such that the imaging elements are allsubstantially coplanar or non-coplanar, or some fraction of the imagingelements 200 can be coplanar and another fraction can be non-coplanar,or any combination of coplanar and non-coplanar. The supports canposition the imaging elements 200 such that the optical axes of theimaging elements 200 are all substantially parallel, only a portion aresubstantially parallel, or none are substantially parallel. The imagingelements 200 can be mounted in such a way that, in a direction movingaway from the imaging sensor 220, their optical axes converge, diverge,converge along one axis and diverge along another axis, converge alongone axis and are substantially parallel along an orthogonal axis, ordiverge along one axis and are substantially parallel along anorthogonal axis.

Each imaging element 200 includes a sensor 220 and an optical system210, which can include one or more lens elements. The field of view ofthe imaging elements 200 used in the imaging system 100 can be differentfrom one another or substantially identical. By combining imagescaptured from one or more of the imaging elements 200, the overallresolution of output images can be increased. Further, the portions ofthe captured images that correspond to the overlapping regions may bestacked to reduce noise. For instance, the noise from overlappingregions can be averaged together in some cases to provide noisereduction.

If the imaging elements 200 capture images that are aligned along pixelboundaries, the overlapping regions may be stacked directly. However, itis likely that the overlapping portions will not align exactly alongpixel boundaries. That is, the portion of the image captured by aparticular pixel in an imaging element 200 will likely not match exactlywith the portion of the image captured by a pixel in an adjacent imagingelement 200. Rather, the portion of the image captured by a particularpixel in an imaging device 200 will likely partially correspond to twoor more pixels in adjacent imaging devices 200.

Imaging elements 200 can include an optical low-pass filter to avoidaliasing effects that could occur by sampling an image at the resolutionof the sensor. Assuming the optical low-pass filter blocks opticalfrequencies above the Nyquist frequency for a given sensor resolution,the captured image can be up-sampled to increase the number of samplepoints, which allows for more accurate alignment of sample pointsbetween the images. In some cases, a low-pass filter is not used. Forinstance, the low pass filter in some cases can be moved out of theoptical path or can be removed.

Stacking of images can be accomplished in substantially real time by theimaging system 100 having a plurality of imaging elements 200,particularly if information about the alignment of the plurality ofimaging elements 200 is known in advance and/or up-sampling is notneeded. Alternatively, stacking of images can be accomplished, forexample, as part of a post-processing workflow, where more precisealignment of the images could be achieved using more computationallyintensive algorithms.

Similarly, stitching of images can be accomplished in substantially realtime from image data from the plurality of imaging elements 200,particularly if information about the alignment of the plurality ofimaging elements 200 is known in advance and/or up-sampling is notneeded. Alternatively, stitching of images can be accomplished, forexample, as part of a post-processing workflow, where more precisealignment of the images could be achieved using more computationallyintensive algorithms.

The imaging element 200 can include one or more lens components 210 thatfocuses light passing through an aperture onto an array of pixels 220.The lens components 210 can comprise a variety of lens types, and can bemade of a variety of materials including glass and plastic, for example.In some configurations, the lens components 210 are fixed-focal lengthlenses having a relatively wide depth-of-field. The lenses may have afixed zoom ratio as well. In some embodiments, one or more of the lenscomponents 210 includes a liquid lens cell. In some embodiments, thelens components 210 can have a dynamic focus and zoom ratio. In someembodiments, the lens components 210 provide a relatively narrowdepth-of-field. In some embodiments, the lens components 210 includedyes or filters such that substantially monochromatic light or lightwithin a defined portion of the spectrum reaches the sensor 220. Thelens components 210 can be removable, such that differentcharacteristics or properties can be provided to the imaging element 200by changing the lens components 210.

In some embodiments, the imaging elements 200 are configured to have arelatively small size. Keeping the size of the imaging elements 200small can have a number of advantages. For instance, maintaining asmaller form factor can increase the number of imaging elements 200included in the system, providing improved resolution among otheradvantages. Smaller sensors 220 also have reduced noise, improved depthof field, and are less prone to manufacturing defects rates. Also, thenumber of picture elements, or “pixels,” in a sensor typicallydetermines the resolution. A large sensor may use more pixels than asmall sensor. Alternatively, a large sensor may use larger pixels todecrease noise in the image. However, an increase in sensor size meansthere will also be an increase in the likelihood that the sensor willcontain defects. Further, a larger sensor will require a larger lens,with additional expense and weight.

It can be desirable to maintain or improve the larger sensor advantages,such as greater resolution and decreased noise, while minimizing thelarger sensor disadvantages, such as increased overall system physicalsize and cost.

While the size and shape of the sensors 220 and lens components 210 canvary between imaging elements 200, in some embodiments the sensors 220have a substantially square form factor that is about 5 mm by 5 mm, andthe lenses are about 15 mm in diameter. In some embodiments, the sensors220 have a square form factor that is smaller than about 10 mm by 10 mm.In other embodiments, the sensors 220 have a rectangular form factorwith a surface area that is less than about 400 square millimeters, lessthan about 225 square millimeters, less than about 100 squaremillimeters, less than about 50 square millimeters, or less than about25 square millimeters. In various embodiments, the diameter or width ofthe lens components 210 is less than about 50 mm, less than about 40 mm,less than about 30 mm, less than about 20 mm, less than about 15 mm,less than about 10 mm, or less than about 5 mm.

The imaging elements 200 can also optionally include a neutral densityfilter for exposure control, black shading, noise correction and thelike. In another embodiment, a common neutral density filter is used forall of the imaging elements 200.

Sensors 220 may include, for example, an array of charge-coupled devices(CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) image sensorcells, such as active-pixel sensor cells. Such image sensors aretypically built on silicon chips and may contain thousands or millionsof image sensor cells.

The sensor 220 further includes output circuitry 230 configured toprocess and output image information for one or more pixels. Forexample, the output circuitry 230 is configured to process and digitizethe analog pixel values received from the pixel array. The outputcircuitry 230 of sensor 220 in one configuration includes sets ofprogrammable-gain amplifiers (PGAs) and analog-to-digital converters(ADCs), although a variety components may be used in variousimplementations. The output circuitry 230 presents the digitized,processed values of the currently selected set of pixels (e.g., aselected row or subset of rows) for storage and/or further processing.For example, the sensor 220 may transmit the values to a memory, imageprocessing module, or other component of the imaging system 100 forstorage and/or processing. In some instances, the sensor 220 buffers thevalues for one or more rows before transmission. Depending on theembodiment, the output circuitry 230 can be configured to process andoutput a single row of pixels or a subset of two or more rows of pixelsat a given time. In one embodiment, the sensor 220 outputs two rows at atime and can include two instances of similar output circuitry 230.

In some embodiments, one or more imaging elements 200 can include anoptical sensor 220 and lens configuration 210 with additional lenselements near the sensor, such as a micro-lens array. Configured in thismanner, the imaging element 200 can be used to capture image data thatcan be used to produce a light field representation for a portion of theimaged scene, which can be used by the imaging system 100, the imageprocessing system 102, and/or a post-processing system to adjust focus,depth of field, or provide other effects.

The imaging elements 200 can be heterogeneous. In one embodiment, theimaging elements 200 use different color filters. For example, one ormore imaging elements 200 may capture light in the green range, one ormore other imaging elements 200 may capture light in the red range, andyet one or more other imaging elements 200 may capture light in the bluerange. In another embodiment, the imaging elements 200 use differentmonochromatic sensors sensitive to a portion of the visible, infrared,or ultraviolet spectrum. The result being similar to the use ofdifferent color filters.

In one embodiment, the imaging system 100 includes a first set ofimaging elements 200 of fixed optical power, and a second set of imagingelements 200 having a variable optical power (e.g., liquid lens cells).In one embodiment, the imaging system 100 includes a first set ofimaging elements 200 including liquid lenses, and a second set ofimaging elements 200 including non-liquid lenses (e.g., solid glass orplastic).

In one embodiment, the imaging elements 200 use different pixel sizes.For example, one or more imaging elements 200 may capture light in thegreen range using normal sized pixels, one or more other imagingelements 200 may capture light in the green range using larger pixels,one or more other imaging elements 200 may capture light in the redrange using normal sized pixels, and yet one or more other imagingelements 200 may capture light in the blue range using normal sizedpixels. In another embodiment, one or more imaging elements 200 maycapture light using a Bayer pattern with the sensor having normal sizedpixels, while one or more other imaging elements 200 may capture lightusing a Bayer pattern or light in the green range using larger pixels.The larger pixels tend to have less noise and may provide greaterdynamic range, while the smaller pixels provide greater resolution.These descriptions are intended to be exemplary only, and othervariations of color patterns and pixel sizes are within the scope of theinvention.

In one embodiment, the imaging elements 200 use different sensor sizes.In one embodiment, imaging elements 200 use sensors having differentaspect ratios.

The imaging elements 200 can further have differently sized aperturesand/or focal lengths. For instance, a first group of one or more imagingelements 200 can have a first aperture size and/or focal length, and asecond group of one or more imaging elements 200 can have a secondaperture size and/or focal length. In one embodiment, imaging elements200 in the first group have relatively wide apertures and capture imageswithin a relatively short depth of field, while imaging elements 200 inthe second group have relatively narrow apertures, and capture imageswithin a relatively large depth of field. Depending on the desiredphotographic effect, image data from the first group, the second group,or a combination thereof can be utilized as appropriate. Similarly, theimaging elements 200 having different focal lengths can be used toprovide focus control. The use of a depth map to utilize imagingelements 200 having different apertures and/or focal lengths isdescribed in greater detail herein.

In yet other embodiments, a first group of one or more imaging elements200 can include a lens element(s) 210 made of a first material (e.g.,glass), and a second group of one or more imaging elements 200 caninclude a lens element(s) 210 made of a second material (e.g., plastic).Or a first group of imaging elements 200 may include a lens element(s)210 having a first type of coating (e.g., a first type ofanti-reflective coating) applied thereon, while a second group ofimaging elements 200 may include a lens element(s) 210 having a secondtype of coating (e.g., a second type of anti-reflective coating) appliedthereon.

In addition, the imaging elements 200 can be configured to capture imagedata having different ranges of wavelength and frequency. For instance,particular groups of imaging elements 200 can be configured to imagedata in one or more of the visible, infrared, ultraviolet, x-ray and/orultraviolet spectrums.

In certain cases, the imaging system 100 can exploit the multi-imagingelement configuration by calibrating the imaging elements 200 in theimaging system 100 using the other imaging elements 200. For instance,one or more test images can be captured, and the results can benormalized or otherwise processed to diagnose the individual imagingelements 200. Based on the processing, the imaging system 100 cancalibrate the individual imaging elements 200 in the array to adjustcolor, white balance, camera response curves, and the like. In somecases, instead of, or in addition to, calibrating the individual imagingelements 200, the calibration results are used in post-processing toadjust the captured image data.

In some embodiments, the imaging system 100 includes a controlconfigured to adjust an angular relationship between imaging elements200. The control can be configured to adjust a direction of the opticalaxis, focal length, aperture, zoom ratio, or the like for individualimaging elements 200.

Imaging Blocks or Facets

FIG. 3 illustrates an example of an imaging block or facet 300 of animaging system 100, such as a dense field imager. The facet 300comprises a plurality of imaging elements 200 with overlapping fields ofview. The facet 300 includes an element support 302 configured tosubstantially secure the imaging elements 200 in a particular positionand/or orientation. The facet 300 includes facet acquisition module 304configured to receive data from the imaging elements 200. In someembodiments, the facet acquisition module 304 receives raw pixel datafrom the imaging elements 200. In some embodiments, the facetacquisition module 304 receives processed image data from the imagingelements 200, such as compressed image data.

The facet acquisition module 304 can be a common acquisition board forall imaging elements 200 on the facet 300. For example, the facetacquisition module 304 can include a FPGA configured to receive imagedata from the imaging elements 200 and output processed image data forthe dense field acquisition module 118 of the dense field imager 100.

The imaging elements 200 can be mounted on an element support 302 wherethe element support 302 can be substantially rigid or it can beflexible. For example, the element support 302 can change shape toaffect an overlapping of the fields of view of the imaging elements 200,such as becoming convex to decrease the overlap between fields of viewof adjacent imaging elements or becoming concave to increase overlap.The element support 302 can have a variety of shapes such as planar,concave, convex, or some other regular or irregular configuration. Theelement support 302 can be configured to emphasize a desirable output,such as increasing an overlap of the fields of view of imaging elements200 where it is desirable to fuse image data to increase resolution,decrease noise, increase position accuracy, and the like. The elementsupport 302 can be configured to decrease an overlap of the fields ofview of adjacent imaging elements 200 to increase a composite field ofview of the facet 300. The element support 302 can be flat, arched, orsome other geometric or non-geometric shape.

The imaging elements 200 can be arranged in any particular fashion onthe element support 302. For example, the imaging elements 200 can be inrows, columns, staggered, in a circular pattern, in a randomconfiguration, an irregular configuration, in a circular or hexagonalconfiguration, or the like.

In some embodiments, the imaging elements 200 can have heterogeneousproperties configured to enhance particular aspects of the dense fieldimager 100. For example, the imaging elements 200 can be configured tobe smaller to increase a density of imaging elements 200 on the facet,to decrease a size of the dense field imager 100, to increase sharpnessof an image, and the like. As another example, the aspect ratios of theimaging elements 200 can be configured to provide a desired compositefield of view. The image sensors can be rectangular and can be orientedsuch that the longer dimension is substantially vertical rather thanhorizontal, which is a typical orientation for an image sensor whenacquiring video, for example. The image sensors can be a mix of square,circular, and/or rectangular shapes. The imaging elements 200 can beoriented pointing in different directions. The imaging elements 200 canbe configured to have different focal lengths, resolutions,sensitivities, etc. Differences in imaging elements 200 can be accountedfor in image processing at the facet acquisition module 304, the densefield acquisition module 118, or the dense field processing module 108.

FIG. 4 illustrates an example embodiment of a facet 300 comprising aplurality of imaging elements 200 coupled together on a facet support302. The example facet 300 comprises 16 imaging elements 200 arranged ina 4×4 grid. Different numbers and configurations of imaging elements 200are possible, and in some configurations the number of imaging elements200 per facet 300 is different. The facet 300 is electrically coupled toa facet acquisition board 306, wherein the facet acquisition board 306is configured to receive image data from the image sensors 200 and/ordata from facet acquisition modules as described above with reference toFIG. 3. In some embodiments, the facet acquisition board 306 isconfigured to receive data from individual imaging elements 200 andcombine the data into a combined output. In some embodiments, the facetacquisition board 306 is configured to receive data from each imagingelement 200 and combine the data into a light field representation.Combining image data can include any of the image processing functionsdescribed herein, including without limitation, fusing, stitching,enhancing resolution, producing light field data, and the like. Asconfigured in the illustrated example of FIG. 4, facets 300 can beconfigured to be joined together in a modular fashion, such that amodular imaging system can be built using multiple facets groupedtogether, e.g., to form a modular dense field imager 100.

Data acquired and processed on the facet acquisition board 306 can bestored on the board 306 and/or it can send the data to another location.The facet acquisition board 306 can be configured to send the data overa wired connection or over a wireless connection. Wired connections caninclude any cabled connection including, without limitation, USB cables,FireWire, Ethernet, serial cables, coaxial cables, twisted pairs, customcabling, or the like. Wireless data transmission can be accomplishedusing radio frequency techniques including 802.11 standards (e.g., WiFiand/or WiGig), Bluetooth®, CDMA, GSM, NFC, or the like. The faceacquisition board 306 can include one or more antennas and one or moretransceivers configured to send and receive data. In some embodiments,the wired and/or wireless transfer rates can exceed 20 Gbps, 10 Gbps, 5Gbps, 1 Gbps, or 500 Mbps. Using parallel wireless transmissiontechniques with multiple frequency bands and high speed protocols canenable the facet 300 and/or groups of facets to send all acquired dataat an acquisition frame rate to processing and/or storage systems thatare remote from the facet acquisition board 306. In some embodiments,the acquisition frame rate can be at least about 240 fps, at least about5 fps and/or less than or equal to about 240 fps, at least about 10 fpsand/or less than or equal to about 120 fps, at least about 24 fps and/orless than or equal to about 60 fps, at least about 30 fps and/or lessthan or equal to about 40 fps, or less than or equal to about 5 fps.

The facet acquisition board 306 can include one or more computerprocessors, FPGAs, ASICs, DSPs, or the like configured to process imagedata. The components of the facet acquisition board 306 can beconfigured to receive image data from a plurality of imaging elements200 and output light field data. The components of the facet acquisitionboard 306 can be configured to receive pixel data from imaging elements200 and output light field data. In some embodiments, the facetacquisition board 306 provides some preliminary image processingfunctionality and sends the output of its processing to a processingstation for editing, coloring, re-focusing, etc.

In some embodiments, each image facet 300 includes acquisitionelectronics (e.g., computer processors, ADCs, ASICs, FPGAs, DSPs, etc.)configured to collect image data from the imaging elements 200associated with the facet 300. The acquisition electronics can beconfigured to pass the image data to the facet acquisition board 306, toanother image facet acquisition electronics (e.g., chaining togetherdata collection for image facets), to a remote system (e.g., a systemthat is not physically coupled to the dense field imager but that may bewired or wirelessly connected to the dense field imager for datatransfers), or the like. In some embodiments, the facet acquisitionboard 306 includes electronic elements configured to receive and/orconsolidate output from one or more imaging elements 200. For example,the facet acquisition board 306 can be configured to receive a firstimage data set representative of a first portion of a light field, toreceive a second image data set representative of a second portion of alight field, and to combine the first and second image data sets to forma third image data set wherein the third image data set can includelight field data, combined image data, dense field data, a dense fieldimage data set, a spatial relationship tensor, a 5 dimensional tensorrepresentative of a light field, a tensor having five or more dimensionsrepresenting a combined light field representation and information forgenerating viewpoints from the combined light field representation,and/or one or more images.

FIGS. 5A and 5B illustrate schematic drawings showing a dense fieldimager 100 with an array of imaging elements 200 coupled to a commonsupport and acquisition board 505. As illustrated, the dense fieldimager 100 includes an array of 192 imaging elements 200, the arrayhaving 8 rows and 24 columns. Other configurations are possibleincluding different numbers of imaging elements 200 (e.g., less than 10,more than 10, more than 20, more than 50, more than 100, more than 200,more than 500, less than 500, etc.) and different arrangements ofimaging elements 200 (e.g., a regular array, a staggered array,circular, hexagonal, irregular or random distribution, etc.). Theexample embodiment illustrated in the schematic of FIG. 5A includespassive cooling elements 510 configured to provide cooling to theprocessing and acquisition electronics included on the common supportand acquisition board 505. The example embodiment illustrated in theschematic of FIG. 5B includes data acquisition and processing elements515 at various locations on the common support and acquisition board505. As described herein, the data acquisition and processing elements515 can be configured to acquire image data and/or to generate lightfield data and to output combined image data, light field data, combinedlight field data, and/or dense field data.

In some embodiments, the array of imaging elements 200 on the commonsupport and acquisition board 505 can be logically and/or physicallydivided into facets. The facets can include 2 or more imaging elements200. The multiple facets on the common support and acquisition board 505can be part of a modular dense field imager 100, where multiple arraysof imaging elements 200 can be joined to form a larger dense fieldimager 100. For example, a dense field imager 100 can include two ormore arrays of imaging elements 200 mounted to different common supportand acquisition boards 505. The common supports 505 can be physicallyjoined together or they can be physically separate but configured tosend their data to a common processing system. In some embodiments, animage processing system can receive data from a plurality of modulardense field imagers 100 and combine the data to produce dense fieldimage data.

The common acquisition board 505 can be configured to combine image datain a staged fashion. For example, the common acquisition board 505 canreceive image data from imaging elements 200 in a facet and combine theimage data to form a light field representation. Adjacent light fieldrepresentations can then be combined to form dense field image data.This process can proceed until image data from substantially all imagingelements 200 has been combined into a dense field representation.

In some embodiments, a subset of the imaging elements 200 on the commonsupport and acquisition board 505 are dedicated to monitoring the densefield imager 100. In some embodiments, the rest of the imaging elements200 are used to generate light field data or dense field data. In someembodiments, the imaging elements 200 that are used to monitor thesystem can change according to monitor criteria. The monitor criteriacan include criteria regarding a quantity of image data to be producedfor use in creating light field data, a quality factor for image data,and the like.

Field of View of a Dense Field Imager

The field of view of a dense field imager 100 is a result of the fieldsof view of the component facets 300, and the fields of view of thecomponent facets 300 are a result of the fields of view of the imagingelements 200 on a facet 300. Fields of view of imaging elements 200 andfields of view of facets 300 can be configured to substantially overlapand/or to overlap primarily along neighboring or adjacent edges. Forexample, imaging elements 200 in a facet 300 can be configured to havesubstantially overlapping fields of view to facilitate the generation oflight field data, and facets 300 in a dense field imager 100 can beconfigured to have fields of view that primarily overlap along the edgesto increase a resulting field of view of a dense field representation.Other configurations are possible as well, such as having imagingelements 200 whose fields of view overlap primarily along the edges andfacets 300 whose fields of view substantially overlap. It may beadvantageous, in some embodiments, to have imaging elements 200 andfacets 300 whose respective fields of view substantially overlap withone another to produce relatively high quality dense field data. It maybe advantageous, in some embodiments, to have imaging elements 200 andfacets 300 whose respective fields of view overlap with one anotherprimarily along adjacent edges to produce dense field data having arelatively large field of view.

FIG. 6A illustrates a field of view for an individual imaging element200, such as an imaging element 200 in a dense field imager 100. Thelens field of view 212 is larger than the sensor field of view 214 thatis actually captured by the sensor 220. Only the field of views aredepicted in FIG. 6A. Physical components such as a physical sensor 220or lens 210 are not shown in FIG. 6A.

FIGS. 6B and 6C illustrate overlapping fields of view for example arraysof imaging elements 200 that are arranged in rectangular and hexagonalgrids, respectively. The imaging elements 200 can be configured in anynumber of ways including, for example, in rectangular and hexagonalgrids (as shown), along concentric circles, staggered, randomdistributions, grouped in facets with the facets arranged in any of thepreceding configurations, etc. As shown, the fields of view 212 for thelenses and the fields of view 214 for the sensors overlap with oneanother, providing some commonality in image scene content captured byneighboring imaging elements 200. Again, physical components such as aphysical sensor 220 or lens 210 are not shown in FIG. 6B or 6C. Theimaging elements 200 themselves may be physically spaced from oneanother, although the fields of view overlap. For example, the imagingelements 200 can be part of a dense field imager 100. The imagingelements 200 can be substantially coplanar with substantially paralleloptical axes, non-coplanar with substantially parallel optical axes,substantially coplanar with non-parallel optical axes, and/ornon-coplanar with non-parallel optical axes. The overlapping fields ofview represented in FIGS. 6B and 6C can represent the overlap at a givendistance from the imaging elements 200. In some embodiments, imagingelements 200 that form a facet 300 can be co-planar with paralleloptical axes and facets 300 that form a dense field imager 100 can benon-coplanar and/or have non-parallel optical axes. In some embodiments,the imaging elements 200 can have fields of view that differ from oneanother. For example, a first imaging element 200 can have a relativelywide field of view and an adjacent imaging element 200 can have arelatively narrow field of view. Imaging elements 200 on a facet 300 canhave similar fields of view or they can cover a range of fields of viewto provide different coverage of the image scene. Facets 300 can havediffering fields of view as well such that a dense field imager 100 canhave facets with non-uniform fields of view. This can increase themodular capabilities of the dense field imager 100 as it can utilize anumber of different types of imaging elements 200 and/or facets 300,allowing for a number of different configurations that can be tailoredfor a desired or particular purpose.

As shown, the overlap can occur primarily in edge regions of the sensor.In other embodiments, the overlap can also occur in the center regionsof the sensor. Limiting the overlap to primarily in the edge regions ofthe sensor provides for a larger effective field of view and/or moreresolution given a particular set of sensors, while overlapping towardsthe center regions of the sensor provides for improved fusion,stitching, and/or noise reduction of images. In some embodiments, theoverlapping image content can be used to reduce the effects ofvignetting or lens distortions, which can typically be more pronouncedin edge regions of the sensor.

In some embodiments, a facet 300 can be configured to have imagingelements 200 whose fields of view substantially overlap. This can beuseful when combining the image data from the imaging elements 200 toform a light field representation. The resulting field of view of thelight field representation can be the total area covered by two or moreimaging elements 200, the total area covered by substantially all or allof the imaging elements 200 of the facet 300, or an area substantiallylimited to where two or more imaging elements 200 have overlappingfields of view.

In some embodiments, image data from facets 300 of a dense field imager100 can be used to generate light field representations whose fields ofview substantially overlap. With substantially overlapping fields ofview, the dense field imager 100 and/or image processing systems can usethe overlapping light field representations to produce dense field datawhose resulting field of view can be an area covered by all orsubstantially all of the facets 300 in the dense field imager 100. Thequality of the resulting dense field data can be increased based atleast in part on the quantity of information available over the combinedfield of view. An increase in the quality of the dense field data cancorrespond to a reduction in noise, an increase in resolution, adecrease in uncertainties related to luminance or radiance values,improved spatial relationship tensors which can be used to generatehigher quality viewpoints or images, and the like.

In some embodiments, image data from facets 300 of a dense field imager100 can be used to generate light field representations whose fields ofview overlap primarily along the edges of their respective fields ofview. In these embodiments, the field of view of a resulting dense fieldrepresentation can be increased or maximized, covering a relativelylarge area. This can be advantageous where it is desirable to acquire,with a relatively small number of imaging elements 200 or facets 300,dense field data having a wide field of view.

FIGS. 7A to 7F illustrate a plurality of imaging elements 200 a, 200 b,200 c that are spaced on a variety of surfaces and that have a varietyof fields of view. In some embodiments, the imaging elements 200 a, 200b, 200 c are part of a facet 300 on a dense field imager 100. Althoughthe description that follows discusses the effects of theseconfigurations with respect to imaging elements 200, the same discussioncan be applied to the fields of view of facets 300 of a dense fieldimager 100. For example, rather than illustrating imaging elements 200a, 200 b, 200 c, FIGS. 7A to 7F could illustrate a plurality of facets300 a, 300 b, 300 c that are spaced on a variety of surfaces and thathave a variety of fields of view. The orientation and fields of view ofthe imaging elements 200 a, 200 b, 200 c can be configured to enhance oroptimize generation of light field data. Similarly, the orientation andfields of view of facets 300 in a dense field imager 100 can beconfigured to enhance or optimize generation of dense field data.

FIG. 7A illustrates a plurality of imaging elements 200 a, 200 b, 200 cthat are spaced on a planar surface. The cross-hatched region is theregion of overlap for imaging elements 200 a and 200 b. To determineand/or control where the overlap region occurs, the distance betweenimaging elements, d, and opening angles, θ, can be used. For example,for imaging elements 200 a and 200 b with substantially the same openingangle, θ (which is half of the field of view and is measured from theoptical axis OA to the outer edge of the field of view), the distance tothe overlap region, l, is equal to d/(2*tan(θ)). The area of overlap ata distance l+x from the imaging elements 200 a, 200 b is calculatedbased on the similarity of the triangles (with the first triangle beingdefined as the triangle with d as the base and l as the height and thesecond triangle the overlapping region). At the distance l+x from theimaging elements, the cross-sectional area of overlap is equal to0.5*d*(x/l).

FIG. 7B illustrates a plurality of imaging elements 200 a, 200 b, 200 cthat are spaced on a concave surface. The field of view of the imagingelements 200 a, 200 b, 200 c substantially overlap. A geometricalcalculation that is similar to that described with reference to FIG. 7Ais shown in this figure for the overlapping region. Here, because theimaging elements are mounted on a concave surface, an additionalparameter is used. The angle α is used to represent an offset angle ofthe optical axis compared to an axis that bisects the distance betweenthe imaging elements and intersects with the optical axes OA_(a) andOA_(b) where they intersect. Another way to understand this axis is thatit represents a combined or average optical axis for imaging elements200 a and 200 b. The distance, l, to the intersection of the fields ofview is then d/(2*tan(θ+α)). The area of overlap at a distance l+x alongthe combined optical axis is 0.5*d*(x/l), if the respective outer fieldsof view diverge or are parallel (e.g., θ≥α). If the respective outerfields of view converge (e.g., θ<α), the area is a boundedquadrilateral.

FIG. 7C illustrates a plurality of imaging elements 200 a, 200 b, 200 cthat are spaced on a convex surface. The field of view of the imagingelements 200 a, 200 b, 200 c are substantially adjacent. The calculationof the overlapping region here proceeds as with that shown in FIG. 7B,with the distance, l, being d/(2*tan(θ−α)). The area of overlap at adistance l+x along the combined optical axis is 0.5*d*(x/l). Thecondition that is satisfied to provide an overlapping region is θ>α.

FIG. 7D illustrates a plurality of imaging elements 200 a, 200 b, 200 cthat are spaced on a planar surface, with a decrease in the angularview.

FIG. 7E illustrates a plurality of imaging elements 200 a, 200 b, 200 cthat are spaced on a concave surface, with a decrease in the angularview. The field of view of the imaging elements 200 a, 200 b, 200 csubstantially overlap. Because of the decrease in angular view, theradius of the concave surface increases as compared to the concavesurface of FIG. 7B.

FIG. 7F illustrates a plurality of imaging elements 200 a, 200 b, 200 cthat are spaced on a convex surface, with a decrease in the angularview. The field of view of the imaging elements 200 a, 200 b, 200 c aresubstantially adjacent. Because of the decrease in angular view, theradius of the convex surface increases as compared to the convex surfaceof FIG. 7C.

FIGS. 7C and 7F illustrate that varying the alignment of a plurality ofimaging elements in a convex fashion can provide increased resolution,even if the field of view changes.

FIGS. 7A and 7D illustrate that even when the alignment of a pluralityof imaging elements does not change, using the plurality of imagingelements provides for increased resolution and reduced noise, even ifthe field of view changes.

The imaging elements 200 may be mounted, for example, on a flexiblesurface or membrane. The surface may adjust with adjustments to the zoomsettings of the imaging elements 200. For example, the flexible surfacemay adjust to provide substantial overlap in the field of view betweenimaging elements 200. As another example, the flexible surface mayadjust to keep the field of view between imaging elements 200substantially adjacent, allowing for some overlap to stitch the imagestogether. The configurations illustrated in FIGS. 7A to 7F can each beaccomplished in separate devices, or in a single, adjustable device, ifthe sensors are mounted to a support such as a membrane which is capableof a tympanic displacement. The sensors can be positioned on a membraneradially symmetrically about a displacement axis which is coincident orparallel to the primary viewing axis of the array. Slight displacementof the membrane along the displacement axis, away from the field of viewwhile restraining the periphery of the membrane, will deflect themembrane into a concave configuration as seen in FIG. 7B. Displacementof a point on the membrane along the displacement axis in the directionof the field of view will produce a convex configuration as seen in FIG.7C. Displacement can be controllably accomplished in any of a variety ofways, such as by magnetic displacement, acoustical displacement or avariety of mechanical or electromechanical systems for providing smallmechanical displacement in response to a control signal.

In some cases, the distance between sensors can vary. For example, anarray of 4×4, 2×2, 6×6, 10×10, 18×9, 6×3, or 10×5 (horizontal×vertical)imaging elements 200 may be arranged so that the spacing between sensorsis approximately ¼ inch between sensor centers, approximately ½ inchbetween sensor centers, approximately one inch between sensor centers,approximately two inches between sensor centers, or approximately fourinches between sensor centers.

The imaging elements 200 may also be mounted on a framework thatprovides for variable distance between sensors. For example, an array of4×4, 2×2, 6×6, 10×10, 18×9, 6×3, or 10×5 (horizontal×vertical) imagingelements 200 may be arranged so that the spacing between sensors adjustsfrom approximately one inch between sensor centers to approximately fourinches between sensor centers. In other cases, the spacing can beadjusted from approximately ¼ inch between sensor centers toapproximately ½ inch, approximately 1 inch, or approximately 4 inchesbetween sensor centers. In yet further embodiments, the spacing can beadjusted from approximately ½ inch between sensor centers toapproximately 1 inch, or approximately 4 inches between sensor centers.

While FIGS. 7B, 7C, 7E, and 7F illustrate arranging the imaging elements200 a, 200 b, 200 c in a concave or convex fashion, other opticaladjustments could be used to obtain the desired alignment. For example,U.S. Patent Publication 2009/0141352 titled “Liquid Optics ImageStabilization,” the entirety of which is incorporated by referenceherein and is included in the attached Appendix, describes the use ofliquid lens cells to stabilize an image along one or more axis. Theimage stabilization described in that publication could be adapted toprovide image alignment of the plurality of imaging elements asillustrated in FIG. 7A, thus avoiding the need for physically aligningthe imaging devices in a concave or convex manner as illustrated inFIGS. 7B, 7C, 7E, and 7F.

FIGS. 7B and 7E illustrate that varying the alignment of a plurality ofimaging elements in a concave fashion can provide for multiple images tobe substantially aligned, even if the field of view changes. The convexand concave configurations shown in FIGS. 7B and 7C can also be used tocounteract the effect of parallax that may result from the spacingbetween the imaging elements. Additionally, in some cases, images can becaptured in sequence from adjacent imaging elements to reduce parallaxissues for video.

Further, imaging elements 200 a, 200 b, 200 c could use liquid lenscells to provide zoom and/or focus, as described in U.S. Pat. No.7,855,838, titled “Liquid Optics Zoom Lens and Imaging Apparatus,” theentirety of which is incorporated by reference herein, and is includedin the attached Appendix.

FIGS. 7A to 7F illustrate imaging elements 200 a, 200 b, 200 c for easeof illustration. However, it is to be understood that multiple imagingelements could be used and not limited in number to three. The imagingelements 200 could be arranged in a variety of patterns, including, forexample, along a linear axis, along a square grid, along a diagonalgrid, along a triangular grid, or in a hexagonal pattern. Thearrangement could be planar, or, for example, concave or convex, orirregular.

FIG. 8 illustrates an example dense field imager 100 having an array ofimaging elements 200 wherein each imaging element 200 includes amonochromatic sensor. The monochromatic sensors can be manufacturedusing techniques described herein. A monochromatic sensor can beconfigured to be sensitive to light falling within a region of theelectromagnetic spectrum, which can span multiple “color” bands. Forexample, a monochromatic sensor can be sensitive to red light, to greenlight, to blue light, to white or broadband light (e.g., light thatcovers all or substantially all of the visible spectrum or light thatcovers from red to blue light), to infrared light, to ultraviolet light,and the like. When used in combination with other monochromatic sensors,the dense field imager 100 can provide dense field data capable ofreproducing a wide variety of colors and light intensities. Using anarray of monochromatic sensors can reduce or eliminate crosstalk betweencolor channels as compared to multi-chromatic image sensors (e.g., aBayer sensor). By reducing or eliminating crosstalk between colorchannels, the sensors can be reduced in size. Reducing the size of theimage sensor can result in improved depth of field and/or increasedperceived sharpness of resolution. The increased sharpness can be basedat least in part on a reduction in interpolation as compared to imagesproduced using traditional multi-chromatic sensors. The manufacture ofmonochromatic sensors is described in greater detail herein.

The monochromatic imaging elements 200 can be configured to haveoverlapping fields of view, as described herein, and overlap regions canbe used to produce color information in an image. For example, in aregion where a red, a blue, and a green imaging element acquire imagedata, an output image can be produced with an appropriate color at eachoutput pixel or position. In some embodiments, a fourth imaging elementcan be included that has a white or broadband sensor (e.g., apanchromatic sensor). The white sensor can provide an ability to controlneutral density and/or dynamic range of a resulting image.

Using monochromatic sensors, color crosstalk can be reduced oreliminated. Color crosstalk can occur where a photon of one colorproduces an electrical output in a pixel of another color (e.g., photonconversion crosstalk), which can occur when a photon strikes a boundarybetween pixels or when a pixel becomes saturated and the signal from thesaturated pixel affects neighboring pixels (e.g., bloom crosstalk). Thebloom crosstalk can be particularly prevalent for white pixels which cansaturate faster than pixels of other colors. In a monochromatic sensor,photon conversion crosstalk is reduced or eliminated, or the effect ofcrosstalk is reduced because pixels are all of the same color. Bloomcrosstalk is reduced or eliminated as well because when one pixelsaturates it is likely that the neighboring pixels of the same colorwill be saturated or nearly saturated as well, reducing the effect ofbloom crosstalk.

Images produced using imaging elements with monochromatic sensors can beimproved by individually controlling sensors. Different color sensorscan be treated differently based on a variety of conditions. Imagingelements can be controlled to have different shutter speeds, apertures,exposures, and the like based at least in part on the color of thesensor. For example, white sensors can be configured to have quickerexposure times to reduce saturation. Color sensors can have longerexposure times to increase photon collection in low-light situations. Aparticular color can have a longer or shorter exposure time for sceneswhere one or more colors may be more prevalent. This feature can be usedto create a dense field imager that does not utilize a traditional Bayerconfiguration, where there are two green pixels for every red and bluepixel. The green imaging element can be configured to have a higherdynamic range, a longer exposure, a wider aperture, or any combinationof these or the like to enhance green photon collection as compared tored and blue imaging elements. In some embodiments, the dense fieldimager 100 includes electronics that control imaging elementsindividually. In some embodiments, the control electronics can beconfigured to treat imaging elements with similar monochromatic sensorsin a similar or uniform fashion.

Example Planar Dense Field Imager

FIGS. 9A to 9C show an example embodiment of a planar dense field imager900 having integrated cooling elements 910 and integrated acquisitionand processing electronics 915. In the illustrated example embodiment,the facets 930 are mounted on a common support and are coplanar, but, insome embodiments, the dense field imager 900 can include facets 930mounted on physically disparate supports and/or supports that arenon-coplanar. In some embodiments, the facets 930 can be mounted onsupports that are non-coplanar and that form planes that are notparallel to one another. As described herein, this can be used to createdesirable overlap in image acquisition and/or increase a field of viewof a resultant dense field representation. The illustrated exampleplanar dense field imager 900 is configured to be a unitary structure,but, in some embodiments, the dense field imager 900 can be configuredto be modular where facets can be added and/or taken away to create avariety of configurations.

The planar dense field imager 900 includes facets 930 where each facet930 includes 16 imaging elements 920, arranged in a 4×4 grid. Otherconfigurations are possible as well, including varying the number andarrangement of the imaging elements 920. For example, the imagingelements 920 can be arranged in a rectangular or hexagonal grid, alongconcentric circles, in an irregular or random distribution, or anycombination of these. The number of imaging elements 920 on a facet 930can be can be at least 2 and/or less than or equal to 400, at least 3and/or less than or equal to 200, at least 4 and/or less than or equalto 100, or any number between 2 and 400 (e.g., 2, 3, 4, 5, 6, 7, 8,etc.). The number of imaging elements 920 per facet 930 can vary fromfacet to facet as well.

The planar dense field imager 900 includes a grid of 12 facets 930,arranged in a 6×2 grid. As illustrated in this example, the facets 930can be configured to be arranged in a regular rectangular grid, butother configurations are possible as well. For example, the facets 930can be staggered, they can be configured as hexagonals and arranged on ahexagonal grid, the facets 930 can be distributed according to anyregular pattern, the facets 930 can be arranged randomly or irregularly,or any combination of these. In some embodiments, the facets 930 arearranged to form a square or a rectangle whose width is shorter than theheight.

The example planar dense field imager 900 is approximately 36 cm wide by26 cm high by 6 cm deep, including the cooling elements 910 and theintegrated electronics 915. The dense field imager 900 can havedifferent sizes and can be larger or smaller than the illustratedexample. For example, the width and/or height of the dense field imager900 can be at least about 6 cm and/or less than or equal to about 100cm, at least about 9 cm and/or less than or equal to about 50 cm, or atleast about 15 cm and/or less than or equal to about 40 cm. In someembodiments, the dense field imager 900 is modular wherein each modulecan have a particular size. The combination of the dense field imagermodules can have any size and/or configuration, based at least upon thedesired application. For example, in some embodiments, a modular densefield imager 900 can be configured to cover a horizontal field of viewof about 360 degrees and/or a vertical field of view of about 180degrees.

Each facet 930 has an imaging area (e.g., the portion of the facet 930with the imaging elements 920) that is approximately 6 cm by 6 cm and anelectronics area (e.g., the portion of the facet 930 with the facetacquisition electronics) that is approximately 6 cm by 8 cm. The imagingarea of the facet 930 can be at least about 1 cm² and/or less than orequal to about 600 cm², at least about 3 cm² and/or less than or equalto about 400 cm², at least about 10 cm² and/or less than or equal toabout 100 cm². In some embodiments, facets 930 can be implemented on awafer and can have appropriate wafer-scale dimensions, as describedherein. The electronics area of the facet 930 can have a variety ofsizes and, in some embodiments, the facet electronics can be integratedinto the dense field imager 900 apart from the facet 930.

In some embodiments, the imaging elements 920 and/or the facets 930 canbe configured to be movable relative to one another. The dense fieldimager 900 can include actuators configured to change a position and/ororientation of individual facets 930 and/or individual imaging elements920. The imaging elements 920 and/or facets 930 can be moved to increaseor decrease an overlap in their fields of view. The movement of theimaging elements 920 and/or the facets 930 can be controlled by anoperator or it can be automatically controlled by the dense field imager900. Movement can be based at least in part on a target field of view, atargeted amount of overlapping image information, a viewing angle, orany combination of these.

Example Non-Planar Dense Field Imagers

FIGS. 10A to 10C illustrate an example embodiment of a dense fieldimager 1000 having a plurality of facets 1030 that are mounted in aconcave fashion. The dense field imager 1000 can include a commonsupport 1005 that lies within an x-y plane (e.g., the plane of the paperin FIG. 10A). The plurality of facets 1030 can be attached or secured tothe common support 1005 through mounting elements 1040.

The orientation of the plurality of facets 1030 will now be describedrelative to a coordinate system having an x-axis and y-axis oriented asshown in FIG. 10A, where the z-axis would extend outward from the page.The top and side views respectively illustrated in FIGS. 10B and 10Cshow the relative configurations of the x-, y-, and z-axes such thatFIG. 10A lies in the x-y plane, FIG. 10B lies in the x-z plane, and FIG.10C lies in the y-z plane. Within this coordinate system, a primaryoptical axis 1002 of the dense field imager 1000 can be defined as aline that is parallel to the z-axis and that intersects the x-y plane inthe geometric center of the plurality of facets 1030, as illustrated bythe circle with a dot. Additionally, a facet plane can be defined as aplane that contains the imaging elements 1020 of that facet 1030. Eachfacet has a facet optical axis 1032 that can be defined as a lineperpendicular to the facet plane that intersects the facet plane in thegeometric center of the facet's imaging elements 1020.

The mounting elements 1040 can be configured to orient one or morefacets 1030 such that the facet optical axes 1032 are not parallel tothe primary optical axis 1002. In the example embodiment illustrated inFIGS. 10A to 10C, the mounting elements 1040 secure the facets 1030 insuch a way that the facet optical axes 1032 generally converge towardsthe primary optical axis 1002 in both the x- and y-directions. Otherconfigurations are possible as well. For example, the mounting elements1040 can be configured to orient one or more facets 1030 such that thefacet optical axes 1032 remain parallel to the primary optical axis 1002in either the x- or y-direction while converging in the other direction(e.g., parallel in the x-direction and converging in the y-direction).The mounting elements 1040 can be configured to orient one or morefacets 1030 such that the facet optical axes 1032 diverge from theprimary optical axis 1002 in either the x- or y-directions whileconverging in the other direction (e.g., diverging in the x-directionand converging in the y-direction). The mounting elements 1040 can beconfigured to orient one or more facets 1030 such that some of the facetoptical axes 1032 remain parallel to the primary optical in a firstdirection while converging in a second, orthogonal direction, otherfacet optical axes 1032 remain parallel in the second direction andconverge in the first direction, other facet optical axes 1032 divergein a first or second direction and converge in the other direction, orany combination of these.

In some embodiments, the mounting elements 1040 generally orient theplurality of facets 1030 along an interior surface of a sphere, anellipsoid, a cylinder, a cylindrical ellipsoid, a paraboloid, ahyperboloid, a polyhedron, or some other regular or irregular surface.In some embodiments, the mounting elements 1040 can be configured to bemovable through mechanical adjustments and/or through actuatorsconfigured to change an orientation of one or more facet planes. Themounting elements 1040 can be manually and/or automatically adjustable.The facets 1030 can be mounted to the common support 1005 to providedesired or targeted overlapping fields of view. The overlapping fieldsof view can be configured to provide image data from a majority ofimaging elements 1020 and/or facets 1030 in a targeted region where onobject or region of interest is located.

FIGS. 10D to 10F illustrate fields of view of some of the facets 1030 onthe dense field imager 1000 illustrated in FIGS. 10A to 10C. FIG. 10Dillustrates the overlapping fields of view of the facets 1030 as viewedin the y-z plane to show the vertical overlapped regions. FIG. 10Eillustrates the overlapping fields of view of the facets 1030 as viewedin the x-z plane to show the horizontal overlapped regions. FIG. 10Fillustrates the overlapping fields of view as viewed in a plane that isnearly parallel to the x-y plane to illustrate the horizontal andvertical overlapped regions. The example dense field imager 1000provides a region of relatively high coverage 1060 where a majority offacets 1030 have overlapping fields of view, as can most easily be seenin FIGS. 10D and 10E.

FIGS. 11A to 11C illustrate an example embodiment of a dense fieldimager 1100 having facets 1130 that are mounted in a convex fashion. Thedense field imager 1100 can include a common support 1105 that lieswithin an x-y plane (e.g., the plane of the paper in FIG. 11A). Theplurality of facets 1130 can be attached or secured to the commonsupport 1105 through mounting elements 1140.

Following the coordinate system convention described with reference toFIGS. 10A to 10C, the mounting elements 1140 can be configured to orientone or more facets 1130 such that the facet optical axes 1132 are notparallel to the primary optical axis 1102. In the example embodimentillustrated in FIGS. 11A to 11C, the mounting elements 1140 secure thefacets 1130 in such a way that the facet optical axes 1132 generallydiverge from the primary optical axis 1102 in both the x- andy-directions. Other configurations are possible as well. For example,the mounting elements 1140 can be configured to orient one or morefacets 1130 such that the facet optical axes 1132 remain parallel to theprimary optical axis 1102 in either the x- or y-direction whilediverging in the other direction (e.g., parallel in the x-direction anddiverging in the y-direction). The mounting elements 1140 can beconfigured to orient one or more facets 1130 such that the facet opticalaxes 1132 converge towards the primary optical axis 1102 in either thex- or y-directions while diverging in the other direction (e.g.,converging in the x-direction and diverging in the y-direction). Themounting elements 1140 can be configured to orient one or more facets1130 such that some of the facet optical axes 1132 remain parallel tothe primary optical in a first direction while diverging in a second,orthogonal direction, some facet optical axes 1132 remain parallel inthe second direction and diverge in the first direction, some facetoptical axes 1132 converge in either the first or second direction anddiverge in the other direction, or any combination of these.

In some embodiments, the mounting elements 1140 generally orient theplurality of facets 1030 along an exterior surface of a sphere, anellipsoid, a cylinder, a cylindrical ellipsoid, a paraboloid, ahyperboloid, a polyhedron, or some other regular or irregular surface.In some embodiments, the mounting elements 1140 can be configured to bemovable through mechanical adjustments and/or through actuatorsconfigured to change an orientation of one or more facet planes. Themounting elements 1140 can be manually and/or automatically adjustable.The facets 1130 can be mounted to the common support 1105 to provide adesired or targeted combined field of view. The combined field of viewcan be configured to provide coverage for a relatively wide area toprovide image information over the area using a single imaging device(e.g., the dense field imager 1100) rather than a multitude ofindividual imaging devices.

FIGS. 11D to 11F illustrate fields of view of some of the facets 1130 onthe dense field imager 1100 illustrated in FIGS. 11A to 11C. FIG. 11Dillustrates the substantially adjacent fields of view of the facets 1130as viewed in the y-z plane to show the vertical overlapped regions. FIG.11E illustrates the substantially adjacent fields of view of the facets1130 as viewed in the x-z plane to show the horizontal overlappedregions. FIG. 11F illustrates the fields of view of the facets 1130 asviewed in a plane that is nearly parallel to the x-y plane to illustratea size of the combined field of view of the dense field imager 1100. Theexample dense field imager 1100 can be configured to provide dense fielddata over a relatively wide field of view.

Example Dense Field Imager with Display

FIGS. 12A and 12B illustrate perspective views of an example dense fieldimager 1200 including a plurality of imaging elements 1220 arranged inan array. The imaging elements 1220 can include a sensor, electronics,and/or an optical system. The imaging elements 1220 are contained withina body 1202. As such, the dense field imager 1200 can be a substantiallyintegrated, portable unit. A first face 1204 of the body 1202 includes atransparent or substantially transparent surface 1206, while a secondface 1208 includes a viewfinder screen 1210.

The imaging elements 1220 are situated within the body 1202 such thattheir respective optical systems gather and focus light that is incidenton and passing through the surface 1206. The surface 1206 can be made ofglass or plastic, for example. In one embodiment, the surface ispartially opaque, and includes transparent windows that correspond toeach of the individual imaging elements 1220. In some cases, the surface1206 is treated with one or more anti-reflective coatings in order toreduce glare or lens flare. The surface 1206 in some cases is a lens,and is shaped to refract light in a desired fashion. In some instances,one or more additional lenses or other optical elements are situatedbeneath the surface 1206, between the optical systems of the individualimaging devices and the face 1204.

Depending on the embodiment, individual imaging elements 1220 in thearray can be adjacent to one another, or can be spaced by an appropriateamount. Moreover, physical barriers can be included between theindividual imaging elements 1220 to reduce cross-talk or other noisebetween adjacent imaging elements.

The display 1210 can include any type of monitoring device. For example,but without limitation, the display 1210 can include an LCD panel. Insome embodiments, instead of or in addition to the integral display1210, a display is connected to an infinitely adjustable mountconfigured to allow the display 1210 to be adjusted to any positionrelative to the body 1202 of the dense field imager 1200 so that a usercan view the display 1210 at any angle relative to the body 1202. Insome embodiments, a separate display can be connected to the body 1202through any type of appropriate cabling or other connection, orwirelessly.

The display 1210 may also include a “look-around” feature, in which thedisplay 1210 shows images for playback that have a larger scene areathan the those that are being recorded for playback and/or editing. Forinstance, the dense field imager 1200 may be capable of capturing imagedata corresponding to a first image scene area but, due to the desiredaspect ratio or other formatting parameters, the system only includesimage data corresponding to a certain sub-set of the captured imagescene area in the recorded video file. In such cases, the system maydiscard a certain amount of edge or boundary content for recordingpurposes. However, the display 1210 may depict some or all of the edgeor boundary content in addition to the recorded content. Moreover, thedisplay 1210 may include a rectangular box or other indication of whatportions of the displayed video are actually being recorded. Oneadvantage of this feature is that camera operators can anticipate whenobjects are about to enter the recorded scene.

The dense field imager 1200 includes a control interface 1212. Thecontrol interface 1212 can include any of a variety of standard userinterface features. For instance, the display 1210 may form part of thecontrol interface 1212 and can be a touch screen, with integratedcontrols in the touch screen. Separate controls 1214 such as one or moreknobs, buttons, keypads and the like may also be used. In oneembodiment, the dense field imager 1200 includes a separate keyboard orother interface hingeably attached to an edge of the body 1202. Thecontrols 1214 can provide a variety of functions including, for example,toggling the dense field imager 1200 between motion and still modes,entering a record mode, operating one or more of the displays or othercomponents of the dense field imager 1200, and the like.

As shown, one or more handles 1216 can be attached to the body 1202.While the illustrated embodiment includes two handles 1216 attached tothe left and right sides of the body 1202, one or more handles 1216 canalso be attached to the top and/or bottom sides of the body 1202. Thehandles 1216 may be releasably attachable to the camera body in somecases. For instance, the handles 1216 can be attached for hand-held use,but removed for studio or other non-hand-held use.

One or more of the handles 1216 can include an interface (not shown)including features for mechanically and/or electrically coupling thehandle to a corresponding interface (not shown) on the body 1202. Thehandles 1216 may be releasably attachable to the body 1202 via a varietyof mechanisms including friction-fit, snap-fit, threaded components, andthe like.

In some embodiments, one or more of the handles 1216 further includes avariety of controls for operating the dense field imager 1200. Thecontrols may include exposure controls, focus controls, and the like,which may be user-definable and suitable for use in still and/or videoapplications. The handles 1216 may be particularly suited for hand-heldand light-weight tripod use. In certain embodiments, one or more of thehandles 1216 includes a rechargeable battery, allowing for lightweightand low-profile remote use without a separate power source. Forinstance, the battery may be releasably insertable into a correspondingreceptacle in the handle 1216. Alternatively, one or more batteries canbe integrated into a handle 1216. When the battery reaches discharge,the handle 1216 can be removed from the dense field imager 1200 andreplaced with a second handle, containing a second, fully chargedbattery.

While three handles 1216 are shown in FIGS. 12A and 12B, other numbersand types of handles can be used. In one configuration, a fourth handleis positioned on the top side of the body 1202, providing convenienthandling regardless of the orientation of the dense field imager 1200.

The dense field imager 1200 can further include one or more ports 1218providing output and/or input connectivity to and from external devices.A wide variety of types of ports can be used, including, withoutlimitation, Ethernet, USB, USB2, USB3, IEEE 1394 (including but notlimited to FireWire 400, FireWire 800, FireWire S3200, FireWire S800T,i.LINK, DV), SATA, SCSI, monitoring ports capable of outputting highresolution image data (e.g., 1080p, 2 k, or 4 k image data), such asHD-SDI, HDMI, etc.

The dense field imager can further include one or more mounting points1219 which serve as mounting points for a variety of components. For,instance, the dense field imager 1200 can be compatible with variousrails, rods, shoulder mounts, tripod mounts, helicopter mounts, matteboxes, follow focus controls, zoom controls, and other features andother accessories known in the art. And one or more of devices providingthese functions can be mountable on one or more brackets or othermounting points on the dense field imager 1200.

In some cases, a remote unit (an example of which is shown in FIG. 13)includes a control interface and is in wireless communication with awireless interface of the dense field imager 1200, providing operatorswith the ability to control camera operation remotely. For example, theremote and the camera may include wireless transceivers. In yet otherconfigurations, the wireless interface is configured to communicate datawirelessly. For instance, image data can be communicated to an externaldevice for remote viewing purposes. This can be particularly useful insituations where the viewfinder 1210 is not visible or accessible to thecamera operator, such as where the dense field imager 1200 is mounted toa vehicle.

As shown, the dense field imager 1200 has a depth d, width w, and heighth, and the form factor is generally configured for straightforwardhandheld and other portable operation. The multi-imaging elementconfiguration allows for a particularly low-profile design, while stillproviding the high performance video and/or still capture describedherein. For instance, the depth d can be less than about 0.5 inches,less than about 1 inch, less than about 1.5 inches, less than about 2inches, or less than about 3 inches, depending on the configuration.

While the imaging elements 1220 are arranged in a 3×6(vertical×horizontal) array in the illustrated embodiment, thearrangement and number of imaging elements 1220 can vary depending onthe embodiment. In another embodiment, the imaging elements 1220 arearranged in a 9×18 (vertical×horizontal) array.

The imaging elements 1220 of the illustrated embodiment can be groupedinto facets where each facet has imaging elements 1220 arranged in a 2×3(vertical×horizontal) array, a 3×3 array, a 2×1 array, or a 1×3 array,or any combination of these. In another embodiment, the number ofimaging elements 1220 can differ and the facets can be configuredaccordingly.

The resolutions of the individual sensors can vary. However, the densefield imager 1200 can be configured to produce an output that is notdependent on the input resolution. For example, the dense field imager1200 can include a plurality of facets, each facet having a plurality ofimaging elements 1220. Each facet can be configured to produce lightfield representations, and the dense field imager 1200 can be configuredto combine the light field representations from the plurality of facetsto produce a dense field representation. From the dense fieldrepresentation, viewpoints can be generated at a variety of outputresolutions which are independent of the input resolution. The above wasdescribed in relationship to spatial resolution, but it also applies fortemporal resolution. Accordingly, the dense field imager 1200 canproduce an output that can be used to generate viewpoints (e.g., imagesand/or video) that whose output resolution (e.g., spatial and/ortemporal resolution) that is not dependent on the input resolution.

The width w of the body 1202 may also be optimized. For instance, thewidth w of the box may be no more than about 20 inches, no more thanabout 18 inches, no more than about 12 inches, or no more than about 14inches. For instance, in certain embodiments, the camera system providesdense field data and has a width w of no more than about 20 inches, nomore than about 18 inches, no more than about 12 inches, or no more thanabout 14 inches, depending on the desired lens size and sensor spacing.In another embodiment, the dense field imager 1200 provides dense fielddata and has a width w of no more than about 10 inches, no more thanabout 8 inches, no more than about 6 inches, or no more than about 4inches, depending on the desired lens size and sensor spacing.

Moreover, the dense field imager 1200 also provides for a particularlylarge total imaging surface area in a low-profile package. For instance,the total span of the sensing surface, which can correspond to the sumsof longest edges of the individual sensors, can be relatively large incomparison to the depth d of the body 1202 along the optical axis. As anexample, where a 9×18 array (vertical by horizontal) of 5 mm×5 mmsensors are used, and the depth d is 25.4 mm (1 inch), the ratio of thetotal span of the sensing area, along the array horizontally, to thedepth d, is about 3.54 (90 mm/25.4 mm). As another example, where a20×40 (vertical by horizontal) array of 5 mm×5 mm sensors are used, andthe depth d is 25.4 mm (1 inch), the ratio of the total span of thesensing area, along the array horizontally, to the depth d is about 7.87(200 mm/25.4 mm). As yet another example, where a 3×6 array (vertical byhorizontal) of 5 mm×5 mm sensors are used, and the depth d is 25.4 mm (1inch), the ratio of the total span of the sensing area, along the arrayhorizontally, to the depth d, is about 1.8 (30 mm/25.4 mm). These arejust a few illustrative examples. Depending on the embodiment, the ratioof the sum of the sensing surface along a particular direction in thesensing plane (e.g., horizontally or vertically across the array ofimaging elements 1220), to the depth d of the body 1220, is at leastabout 0.25, 0.3, 0.5, 0.75, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25,or 50.

The total span of the lens surface can also be relatively large incomparison to the depth d of the dense field imager 1200. For instance,the total span of the lens area can correspond to the sums of the widths(e.g., diameters for circular lenses) of the individual lenses at theirwidest points. As an example, where a 9×18 array of lenses having adiameter of 15 mm is used, and the depth d is 25.4 mm (1 inch), theratio of the span of the lens surface along the horizontal directionacross the array is 10.62 (270/25.4 mm). As another example, where a20×30 array of lenses having a diameter of 15 mm is used, and the depthd is 25.4 mm (1 inch), the ratio of the span of the lens surface alongthe horizontal direction in the array is 17.7 (450/25.4 mm). These arejust a couple of examples. Depending on the embodiment, the ratio of thesum of the total lens surface along a particular direction in thesensing plane (e.g., horizontally or vertically across the array ofimaging devices), is at least about 1, 5, 10, 11, 12, 13, 14, 15, 16,17, 18, 19, 20, 25, 30, 40, 50, 60, 70, 80, 90, or 100.

According to some embodiments, the dense field imager 1200 is modularlyexpandable to increase the number of facets and imaging elements 1220.For instance, the dense field imager 1200 can be extended via attachmentof a dense field imaging module that includes one or more facets, eachfacet having a plurality of imaging elements. In some other embodiments,the dense field imager 1200 may include a slot, bay or other receptaclefor insertion of one or more additional dense field imaging modules. Forinstance, the body 1202 may include a slot on one of its sides forreceiving additional dense field imaging modules. The dense fieldimaging modules may have the same number of facets, imaging elements,and/or a similar form factor as the dense field imager 1200, or can havedifferent numbers of facets, imaging elements, and/or form factors. Inthis manner, the dense field imager 1200 can be scalable to achieve aparticular field of view or other outcome. One or more mountingbrackets, snap-fit or friction-fit mechanisms, or other appropriatecomponents can be used to fasten the dense field imaging modulestogether.

FIG. 12C shows a side perspective view of an embodiment of the densefield imager 1200 having a plurality of imaging elements 1220 withlarge, removable lenses. The imaging elements 1220 can be configured tohave removable lenses such that they can be exchanged and/or replaced.In addition, different types of lenses can be used on different imagingelements 1220. The lenses can also be changed or modified to includefilters to provide desired optical properties. In embodiments such asthose illustrated in FIGS. 12C to 12H, the transparent surface 1206present in the dense field imager illustrated in FIGS. 12A and 12B canbe excluded and/or removed. The relatively large lenses can provide anincreased light sensitivity, increased zoom ratio, increased aperture,and the like when compared to smaller lenses. In addition, using alarger lens and/or imaging element 1220 in general can increase a sizeof the image sensor associated with the imaging element 1220, which canresult in an increase in resolution for an individual imaging element(e.g., more pixels), and/or an increase in light sensitivity (e.g.,larger pixels).

FIG. 12D shows a side perspective view of an embodiment of the densefield imager 1200 having a plurality of imaging elements 1220 withsmall, removable lenses. Similar to the embodiment illustrated in FIG.12C, the small, removable lenses can allow for a variety of lens typesto be included in the dense field imager 1200. The relatively smalllenses can provide for sharper images, larger depths of field, and thelike. In addition, using a smaller lens and/or imaging element 1220 ingeneral can increase the number of imaging elements that can be used ina given space. Moreover, using smaller lenses and/or imaging elements1220 can correspond to using smaller imaging sensors which can produceimages with a higher perceived sharpness.

FIG. 12E shows a side perspective view of an embodiment of a dense fieldimager 1200 having a plurality of imaging elements 1220 withnon-removable, low-profile lenses. The non-removable, low-profile lensescan advantageously reduce a size of the dense field imager 1200, makingit easier to handle and operate. In addition, by making the lensesnon-removable, the dense field imager 1200 can be more resistant tomechanical failures related to the addition and removal of lenses.

FIGS. 12F-H show top perspective views of the dense field imager 1200with a grip attachment 1216, the dense field imager 1200 having aplurality of imaging elements 1220 with, respectively, large, removablelenses; small, removable lenses; and non-removable, low-profile lenses.The grip attachment 1216 can be configured to be removable to allow thedense field imager 1200 to be used in a variety of contexts. The gripattachment 1216 can allow for a user to easily operate and handle thedense field imager 1200, whereas removing the grip attachment 1216 maybe desirable where the dense field imager will be mounted to a mountingdevice or where dense field imaging modules are to be attached. In someembodiments, the grip attachment 1216 includes user interface elementsthat allow the user to operate and configure properties of the densefield imager, such as a zoom, an orientation of imaging elements 1220and/or facets, an acquisition mode (e.g., switching between image andvideo mode), an exposure time, and the like.

FIGS. 12I and 12J show top perspective views of the dense field imager1200, respectively with and without a grip attachment 1216, showing theviewfinder screen 1210. As described above with reference to FIG. 12B,the display 1210 can be used to provide visual feedback to a user of thescene being captured. Because the dense field imager 1200 can be used toacquire light field data and output dense field data, the display 1210can be configured to receive image information from a single imagingelement 1220. In some embodiments, the imaging element 1220 providinginformation to the display 1220 can be automatically selected or it canbe selected by a user. In some embodiments, the imaging element 1220providing image information to the display can be switched according tocriteria and/or user selection. In some embodiments, a facet can provideimage information to the display 1210 and the display can allow a userto interact with light field data provided by the facet. For example,the user can re-focus a scene, zoom in on a scene, change a depth offield of a scene, adjust a viewpoint, and the like based at least inpart on the light field data from a facet. Similar to the imagingelement 1220, the facet can providing data to the display can beselected and/or switched automatically or manually. In some embodiments,the user can use the display 1210 to interact with dense field dataprovided by the dense field imager 1200. In some embodiments, thedisplay 1210 can be used to acquire dense field data, to manipulatedense field data, and/or to configure the dense field imager 1200. Thedisplay 1210 can be configured to display status information to theuser, such as storage information, wireless connectivity information,radiance or luminance information, distances to objects, and the like.

FIG. 13 illustrates an embodiment of a dense field imager 1300 that isconfigured to wirelessly transmit acquired data to an image hub 1360.The dense field imager 1300 can be any of the dense field imagerembodiments described herein, such as those described with reference toFIGS. 1, 5A, 5B, 8, 9A-C, 10A-F, 11A-F, and/or 12A-J. The image hub 1360can be any hardware system adapted to communicate wirelessly with thedense field imager 1300 and to store the data received from the densefield imager 1300.

In some embodiments, the image hub 1360 is portable. For example, theimage hub 1360 can be configured to be worn by a user (e.g., a cameraoperator), attached to the clothing of a user, carried inside a bag by auser, moved with the dense field imager 1300, or the like. In someembodiments, the image hub 1360 is configured to have a substantiallyfixed location and can be located remote from the dense field imager1300. For example, the image hub 1360 can be an image processing systemthat is located at a central image processing location that is within awireless communication distance from the dense field imager. Thewireless communication distance can vary depending on the wirelessprotocol to be used. For example, the wireless communication distancecan be less than or equal to about 500 feet, less than or equal to about100 feet, less than or equal to about 20 feet, less than or equal toabout 10 feet, less than or equal to about 5 feet, less than or equal toabout 2 feet, less than or equal to about 1 foot, or less than or equalto about 6 inches.

In some embodiments, the image hub 1360 is configured to communicatewith the dense field imager 1300 using a wireless protocol capable oftransmitting data at a desired data transmission rate. For example, thewireless communication can be accomplished using 802.11 standards (e.g.,WiFi and/or WiGig), Bluetooth®, CDMA, GSM, NFC, or the like. In someembodiments, the wireless data transmission rates can be greater than orequal to about 20 Gbps, at least about 500 Mbps and/or less than orequal to about 10 Gbps, at least about 1 Gbps and/or less than or equalto about 5 Gbps, or less than or equal to about 500 Mbps.

In some embodiments, both the dense field imager 1300 and the image hub1360 include a plurality of transceivers and antennas configured tocommunicate data in a parallel fashion over a plurality of datacommunication channels. In this way, the data transmission rates can beincreased to accommodate more data such as when the data acquisitionrate of the dense field imager 1300 increases. For example, the dataacquisition rate of the dense field imager 1300 can increase where anacquisition mode switches from a still image acquisition mode to a videoacquisition mode or where the dense field imaging modules are attachedto the dense field imager 1300 thereby increasing the amount of dataacquired.

The image hub 1360 can be configured to have a storage medium forrecording the data received from the dense field imager 1300. Thestorage medium can be any suitable computer storage such as hard diskdrives, solid state drives, or the like. The image hub 1360 can beconfigured to perform image processing as well as data storage. Theimage hub 1360 can include any suitable processing systems such ascomputer processors, FPGAs, DSPs, ASICs, and the like configured toprocess data received from the dense field imager 1300. For example, theimage hub 1360 can be similar to the multi-image processing system 110described herein with reference to FIG. 1.

Multi-Imaging Device Systems

FIG. 14A illustrates a top plan view and FIG. 14B illustrates a sideelevational view of a multi-imaging device system 1400. Themulti-imaging device system 1400 can include two or three or morecameras 1402 that are physically separate from one another, and, in someembodiments, one or more of the cameras 1402 can be a stand-aloneimaging device. For example, the cameras 1402 can include dense fieldimagers, video cameras, cell phone cameras, DSLR cameras, webcams, orany imaging device configured to capture digital image data. The cameras1402 can be different from one another and can have different imagingproperties, such as, for example and without limitation, resolution,acquisition rate, aperture, focus, sensitivity, color gamut, and thelike. The cameras 1402 can be positioned at spaced apart locations andoriented so that a primary optical axis 1403 of each camera is generallydirected toward an object of interest or a common viewing theater. Datafrom the multi-imaging device system 1400 can be transmitted to aprocessing station where image data is combined. The combined image datacan be light field data, dense field data, video data, image data, pixeldata, mathematical representations of image data (e.g., representationsusing fractals or vectors), or any combination of these. The combinedimage data can be used to generate two dimensional or three dimensionalimages, in still images or motion. An example of such a processingstation is described herein with reference to FIG. 1. In someembodiments, simultaneously captured video from one or more of thecameras 1402 in the system 1400 is combined to create a spatial and/ortemporal resolution independent output. For example, each of the cameras1402 can be configured to acquire video data at an acquisitionresolution at an acquisition frame rate where the acquisition resolutionand/or frame rate can be different for each camera 1402. The acquiredvideo data can be combined to generate dense field data that can be usedto generate high-resolution (e.g., 2 k, 4 k, or higher), high-frame rate(e.g., 30 fps, 60 fps, 120 fps, 240 fps, or higher) video output wherethe resolution and/or frame rate of the video output exceeds theacquisition resolution and/or frame rate of any individual camera 1402.For some applications, non-simultaneous data is combined to create atleast a portion of the output video stream.

As discussed, the optical axes 1403 of the cameras 1402 can be directedtowards the common viewing theater. For instance, in certainembodiments, the field of view 1404 of each of the cameras 1402 at leastpartially overlaps with the field of view 1404 of each of the othercameras 1402. In some configurations, the optical axes 1403 of each ofthe cameras 1402 intersect at the same point, or substantially the samepoint, in the viewing theater.

In some embodiments, the cameras 1402 transmit wireless data to one ormore receivers, which route data to a processing station. Cameras 1402can be positioned to surround the viewing theater, and may be attachedto support poles, buildings, trees, or other suitable existing orsupplied support structure. In certain implementations, at least three,and in some applications at least 10, or at least 50 or more cameras1402 are provided in the multi-imaging device system 1400 for imaging aviewing theater.

Due to the disparate arrangement of the cameras 1402 about the viewingtheater, the cameras 1402 can image the viewing theater from generallyany vantage point. In some cases, each of the cameras 1402 is spacedfrom each of the other cameras 1402 by at least 1, 10, 100 or 1000 feet.According to some arrangements, the viewing theater can be delineatedinto halves (e.g., with respect to a top plan perspective and/or sideelevational perspective), and at least one camera 1402 in the array ispositioned in a different half of the viewing theater than at least oneother camera 1402 in the array. Or, the viewing theater may delineatedinto four equally sized segments (e.g., from a top plan perspectiveand/or side elevational perspective), and at least one camera 1402 inthe array is positioned in each of the four segments. In yet furthercases, at least one first camera 1402 a is at least partially facing atleast one other camera 1402 c, such that the other camera 1402 c atleast partly resides in the field of view 1404 a of the first camera1402 a. Depending on the orientation of the cameras 1402, the firstcamera 1402 a can also at least partly reside in the field of view 1404c of the other camera 1402 c.

The cameras 1402 can be positioned about the perimeter of the viewingtheater or, in some cases, one or more of the cameras 1402 are alsopositioned within the viewing theater. Moreover, the cameras 1402 can bearranged in some cases in an irregular manner about the viewing theater.In such cases, the system 1400 can use a depth map to adjust forpositional variations between the cameras 1402 and the viewing theateror portions thereof. This capability gives camera operators flexibilityin placing the cameras 1402 with respect to the viewing theater. Thiscan be particularly beneficial in environments where options for cameraplacement are constrained, such as where it is desirable to hide thecameras. In other cases, the cameras 1402 or a subset of thereof arearranged in a regular manner with respect to one another, e.g., withinthe same plane, on a common hemispherical profile, symmetrically, etc.

As shown in FIG. 14B, the cameras 2 can also be positioned in differentvertical positions with respect to the viewing theater. For instance,the cameras 1402 may be divided into a first plurality, within a firstvertical zone 1405 (e.g., within about 5 or 10 vertical feet) of thelocal ground level. A second plurality of cameras 1402 may be positionedwithin a second zone 1408 (e.g., from about 5 or 10 feet of the groundand about 20 or 30 feet of the ground) vertically above the firstvertical zone 1405, and a third plurality of cameras may be positionedin a third zone 1409, vertically above the second zone 1408. Thevertical separation of cameras may or may not be desired depending uponthe nature of the activity under surveillance in the viewing theater.

The orientation of one or more of the cameras 1402 is adjustableaccording to further aspects. For instance, the cameras 1402 can beoutfitted or mounted on a motor-driven tripod or other apparatus thatallows for user controlled or automated tilt, pan and/or zoom control.In one implementation, such as where the cameras 1402 are in continuousor substantially continuous communication with the processing station,the processing station can maintain up-to-date orientation informationfor each of the cameras 1402. In some embodiments, the processingstation pings the cameras 1402 periodically for status info, or thecameras 1402 periodically broadcast current orientation. According toone method, cameras 1402 report orientation information upon “wake-up”from a power-down or reduced power mode.

All cameras 1402 in the multi-imaging device 1400 may be stationary.Alternatively, a first plurality of cameras 1402 may be stationary andat least one or a second plurality of cameras 1402 may be moving orattached to a structure which is capable of moving (e.g., a vehicle,robot, personnel, drone or other aircraft, satellite). All cameras 1402may be carried by a person or support which is moving or capable ofmoving.

Data from the cameras 1402 can be combined to generate dense field data,light field data, combined image data, or any combination of these orother image representations, as described herein. In some embodiments,combining the data from the cameras 1402 can include determiningcorrespondences between two or more images of approximately the sameoptical target. Determining the correspondences can include registeringthe images and can be intensity-based (e.g., comparing intensitypatterns in a first image to intensity patterns in a second image)and/or feature-based (e.g., finding features in one image and seeing ifthe layout of a subset of features is similar to that in a secondimage). One or more reference points may be identified or introducedinto the theater to facilitate correspondence determination. Forexample, one or more transmitters can be positioned within the theater,for transmitting a signal that can be recorded by the camera andrecognized as a correspondence reference. The transmitted signal can beoptical, RF and, in some environments, acoustical. Active continuous orintermittent transmitters may be desirable in some applications.Alternatively, passive references (e.g., RFID or other system whichresponds to interrogation from the camera) may be used. Optical signalsmay include a continuous or intermittent optical transmitter positionedwithin the theater (UV, visible or infra red) or an optical signal(e.g., laser) transmitted from one or more cameras into the theater.

In general, where image data from multiple cameras 1402 is combined tocreate motion video or still images, any of the techniques describedherein (or other techniques) for stitching together or otherwisecombining the image data can be used. While in some cases, image datafrom all of the cameras 1402 is combined together, data from a selectedsubset of the cameras 1402 in the array can also be used. And, thecurrent selected subset of cameras 1402 can adjust dynamically dependingon the desired effect. For instance, image data from a first subset ofcameras 1402 may be combined to create 2D or 3D image data from a firstperspective, for a first period of time, and then, for a second periodof time, the system combines image data from a second subset of cameras1402, achieving a different viewing theater perspective, and so on. Asone example, referring to FIG. 14A, the system 1400 may combine imagedata from the cameras 1402 a, 1402 b to create video image data from afirst perspective and then transition to combining image data from thecameras 1402 b, 1402 c. The next transition could be to any combinationof cameras (e.g., back to the cameras 1402 a, 1402 b, to the cameras1402 c, 1402 d, to the cameras 1402 a, 1402 b, 1402 c, to all thecameras, etc.). Transitioning to a subset of cameras 1402 having atleast one camera in common with the current subset is preferable in somecases, as it can provide continuity of perspective. However, in somecases, the next subset does not include any cameras 1402 in common withthe current subset.

The mechanism for adjusting the current subset of cameras can vary. Forinstance, a user interface (e.g., touch screen, joystick, voice command,etc.) can provide operators with real-time control of the desiredviewing perspective. In other cases, the system automatically trackscertain image scene objects. As one example, where a person or otherobject of interest is moving through the image scene, the system 1400can dynamically adjust the subset of cameras (or otherwise process thecombined image data) to provide an obstruction-free view of the personor object.

Moreover, in certain cases, to conserve bandwidth or power, such as inlive playback applications, image data for only the selected subset ofcameras 1402 is transmitted for processing. Unselected cameras 1402, insome cases, enter a power-saving mode in which they consume less powerthan the selected subset of cameras 1402.

According to certain aspects, the multi-imaging device system 1400 canalso provide multi-vantage point audio capture capability for theviewing theater. For instance, each camera 1402 may be provided with orbe coupled to a microphone or other audio input device. In such cases,captured audio information associated with each camera 1402 may betransmitted for processing, along with the image data, for example.Instead of or in addition to providing microphones that are integratedwith the cameras 1402, separate microphones can be positioned throughoutthe viewing theater, or otherwise be situated to record soundinformation from the viewing theater.

Similar to the image data, audio information from the cameras 1402 canbe selectively combined. For instance, the system may automaticallyutilize image data captured from one or more cameras in a currentlyselected subset. In other cases, a user may be able to select the audiosource(s). In some cases, a user may select audio data from a singlecamera 1402 or other audio source closest to a particular object orevent of interest, while the selected image data is from a combinationof multiple cameras 1402. Depending on the circumstances, a particularcamera 1402 may be used as an audio source even though it is notcurrently being utilized in the recorded video file (or for playback).

The multi-camera images from the viewing theater can also be combined toadjust the dynamic range of the output video stream (e.g., by tonemapping or other appropriate mechanisms), or to otherwise controlexposure. For instance, lighting conditions may be different fordifferent cameras in array. In one case, highlight detail from one ormore cameras 1402 (e.g., with a higher exposure setting or positioned inrelatively brighter light conditions) is combined with shadow detailsfrom one or more other cameras 1402 (e.g., with a lower exposure settingor positioned in relatively low light conditions). In some cases,lighting conditions for unselected cameras 1402 may be more favorablethan lighting conditions for cameras 1402 in the currently selectedbank. In these and other circumstances, exposure information from one ormore cameras 1402 that are not in the currently selected bank can beused to adjust exposure levels in the output stream. Additionalcompatible high dynamic range techniques are discussed below and aredescribed in U.S. Patent Application Publication No. 2012/0044381entitled “HIGH DYNAMIC RANGE VIDEO”, which is incorporated by referenceherein in its entirety and is included in the attached Appendix.

One or more cameras 1402 in the multi-imaging device system 1400 may becell phone cameras, smart phone cameras, cameras on tablets, hand-heldcameras, or the like. The system 1400 can be configured to receive imagedata acquired using one or more of these disparate cameras and combinethe information to produce video and/or still images from virtualviewpoints or having other characteristics, as described herein. Thus,the multi-imaging device system 1400 can be a dynamic system, with achanging number of cameras 1402 that make up the system 1400. In suchembodiments, the image processing system can be part of a website orother publicly- or privately accessible resource that receives imagedata from users to generate output data. The input data can be used togenerate video and/or still images that incorporate information from oneor more of the cameras 1402 to produce high-quality viewpoints of anobject of interest. For example, multiple people may capture image dataof a newsworthy event, yet the images may be partially occluded,acquired from a disadvantageous viewpoint, shaky, low-resolution,blurry, or the like. By combining the acquired image data from multipleviewpoints using multiple cameras 1402, images and/or video may begenerated that provide an advantageous, high-quality view of the eventof interest. In some embodiments, the acquired image data can becombined and the combined data can be made available for others to useto generate output images and/or video. Software tools can be providedwhich allow users to generate images and video using the output datawhich can be light field data and/or dense field data, for example.

One or more cameras 1402 in the multi-imaging device system 1400 may becarried by one or more geosynchronous satellites, which have theadvantage of remaining permanently in the same area of the sky, asviewed from a particular location on Earth. This allows permanentsurveillance of a desired theater. Thus the multi-imaging device system1400 can include a geosynchronous component or a geosynchronous networkas a portion of the system 1400.

At present, the GeoEye-1 satellite may be the highest resolutioncommercial imaging system and is able to collect images with a groundresolution of 0.41 meters (16 inches) in the panchromatic or black andwhite mode. Spatial resolution is defined as the pixel size of an imagerepresenting the size of the surface area (i.e. m2) being measured onthe ground, determined by the sensors' instantaneous field of view(IFOV). While the satellite is able to collect imagery at 0.41 meters,GeoEye's operating license from the U.S. Government requires re-samplingthe imagery to 0.5 meters for all customers not explicitly granted awaiver by the U.S. Government.

The ground based component of the present multi-imaging device system1400 can add significant depth of data to current GeospatialIntelligence systems. Frequently referred to as GEOINT, GeospatialIntelligence is an intelligence discipline comprising the exploitationand analysis of geospatial data and information to describe, assess, andvisually depict physical features (both natural and constructed) andgeographically referenced activities on the Earth. GeospatialIntelligence data sources include imagery and mapping data, whethercollected by commercial satellite, government satellite, aircraft (suchas Unmanned Aerial Vehicles [UAV] or reconnaissance aircraft), or byother means, such as maps and commercial databases, census information,GPS waypoints, utility schematics, or any discrete data that havelocations on earth. Integration of data from the optical networkdescribed herein can help GEOINT evolve from traditional compilations ofgeospatial information and imagery, towards an emphasis on knowledge.

Depending upon the needs of a particular theater, cameras 1402 can behard-wired or in wireless communication with the data processingstation. A variety of wireless technologies can be used, as describedherein. In some embodiments, the cameras 1402 implement a wireless linkcapable of transmitting relatively high resolution image data, such as adual 2.970 Gigabit HD-SDI wireless link that operates in the 60 GHzfrequency band. Data processing can occur locally, or towns orcontinents away. Cameras 1402 can be powered by ground power, batterypower, solar power or otherwise depending upon a particular assignment.

As discussed, image data from the cameras 1402 can be combined toprovide improved video or still images of select image scene regions.FIG. 14C illustrates a top plan view of an array of cameras 1402recording an object 1406 or other portion of interest in a viewingtheater. As shown, one or more obstructing objects 1407 occlude thefield of view 1404 of the cameras 1402. The processing station canstitch together or otherwise combine the image data from a plurality(e.g., 2, 3, 4, 5 or all) of the cameras 1402 in the array to reduce orremove the occlusions in the combined video output. While the field ofview 1404 of each camera 1402 shown in FIG. 14C is at least partiallyoccluded, in some cases, some of the cameras 1402 have un-obstructedfields of view 1404.

Example Image Processing System

FIG. 15 illustrates a block diagram of an example image processingsystem 1500 for receiving data from a plurality of sources of differenttypes and generating light field data, dense field data, and/orviewpoints from received data. The image processing system 1500 canreceive image data from image data sources 1501 which can include, forexample, a dense field imager 1502, a plurality of imaging devices 1504,a plurality of light field data sources 1506, or any combination ofthese. The image processing system 1500 includes a controller 1508,memory 1510, data storage 1512, a pre-processing module 1514, a lightfield generator module 1516, a dense field generator module 1518, and aviewpoint generator module 1520. The various components of the imageprocessing system 1500 can be configured to communicate with one anotherand/or with external systems through communication bus 1507.

The image data sources 1501 can provide image data to the imageprocessing system 1500. The image data provided to the image processingsystem 1500 can include pixel data from a plurality of image sensors,light field representations, dense field representations, metadata,calibration information, and other information associated with theimages that allow the image processing system to generate light fielddata, dense field data, viewpoints, and the like. The image data sources1501 can include a dense field imager 1502, examples of which aredescribed herein with reference to FIGS. 1, 5A, 5B, and 8-13. The imagedata sources 1501 can include a plurality of imaging devices 1504, asdescribed herein with reference to FIGS. 14A-C. The image data sources1501 can include a plurality of light field data sources 1506, which caninclude, for example and without limitation, imaging devices whichprovide light field data (e.g., plenoptic cameras, light-field cameras,etc.), stored light field data (e.g., computing devices with storedlight field data, external storage with light field data, etc.), densefield imagers that output a plurality of light field representations, orany combination of these.

The image processing system 1500, in some embodiments, is configured toreceive, store, and/or output raw image data. In other embodiments, theimage data is not raw. In some cases, the image processing system 1500stores data received directly from the plurality of sensors, withoutsubstantial modification, substantial processing, or withoutmodification or processing, prior to recording and/or output.

The various modules 1514, 1516, 1518, and 1520 of the image processingsystem 1500 may execute on the controller 1508 residing in the imageprocessing system 1500 and/or may include custom circuitry. One or moremodules may process data that is stored in memory 1510 and/or datastorage 1512, or one or more modules may process data as it comes fromthe image data sources 1501 which can then be stored in memory 1510and/or data storage 1512.

The image processing system 1500 includes the controller 1508. Thecontroller 1508 can include one or more processors and can be used byany of the components of the image processing system 1500, such as thepre-processing module 1514, the light field generator module 1516, thedense field generator module 1518, the viewpoint generator module 1520,to process information. As used herein, the term “processor” refersbroadly to any suitable device, logical block, module, circuit, orcombination of elements for executing instructions. The controller 1508can be any conventional general purpose single- or multi-chipmicroprocessor such as a Pentium® processor, a MIPS® processor, a PowerPC® processor, AMD® processor, ARM® processor, or an ALPHA® processor.In addition, the controller 1508 can be any conventional special purposemicroprocessor such as a digital signal processor. The variousillustrative logical blocks, modules, and circuits described inconnection with the embodiments disclosed herein can be implemented orperformed with a general purpose processor, a digital signal processor(DSP), an application specific integrated circuit (ASIC), a fieldprogrammable gate array (FPGA), or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general purpose processor, such as controller 1508, can be aconventional microprocessor, but the controller 1508 can also be anyconventional processor, controller, microcontroller, or state machine.Controller 1508 can also be implemented as a combination of computingdevices, e.g., a combination of a FPGA and a microprocessor, a pluralityof microprocessors, one or more microprocessors in conjunction with aDSP core, or any other such configuration.

The image processing system 1500 includes memory 1510. The memory 1510can be coupled to the other components of the image processing system1500, such as the controller 1508, data storage 1512, the pre-processingmodule 1514, the light field generator module 1516, the dense fieldgenerator module 1518, and the viewpoint generator 1520. Memory 1510 canrefer to electronic circuitry and/or devices that allow information,typically processing instructions and/or data, to be stored forretrieval for use by the image processing system 1500 in a relativelyquick manner. Memory 1510 can refer to Random Access Memory (RAM) orvarious forms of Read Only Memory (ROM), which are directly connected tothe one or more processors of the image processing system 1500. Othertypes of memory can include bubble memory and core memory.

The image processing system 1500 includes data storage 1512. Datastorage 1512 can be coupled to the other components of the imageprocessing system 1500, such as the controller 1508, memory 1510, thepre-processing module 1514, the light field generator module 1516, thedense field generator module 1518, and the viewpoint generator 1520.Data storage 1512 can refer to electronic circuitry that allowsinformation, typically computer data, to be stored and retrieved. Datastorage 1512 includes computer-readable memory devices and can be in theform of any type of computer-readable medium, such as, for example andwithout limitation, hard drives, solid-state drives, flash memory,optical discs, or any other type of memory device. In some embodiments,the size of the data storage 1512 can be sufficiently large to storelight field data, dense field data, image data, and the like from thevarious modules 1514, 1516, 1518, and 1520. Data storage 1512 can referto external devices or systems or internal devices or components whichare directly connected to the one or more processors of the imageprocessing system 1500.

In some embodiments, the data storage 1512 can be mounted on an exteriorof a camera body (e.g., a dense field imager) or be inserted into a slotor other receptacle in the camera body. Further, in some embodiments,the data storage 1512 can be connected to the other components of theimage processing system 1500 through standard or custom communicationports, including, for example, but without limitation, Ethernet, USB,USB2, USB3, IEEE 1394 (including but not limited to FireWire 400,FireWire 800, FireWire S3200, FireWire S800T, i.LINK, DV), SATA andSCSI. Further, in some embodiments, the data storage 1512 can comprise aplurality of hard drives, such as those operating under a RAID protocol.However, any type of storage device can be used.

Data can be stored in data storage 1512 using any variety of datastorage methods including, without limitation, data stored in databases,as individual files, as a plurality of data files with configuration(e.g., XML files), in a table, index, map, or other data structureincluding metadata associated with the image data. For instance, datastorage 1512 can include raw image data from the dense field imager 1502where the raw image data includes pixel data for imaging elements on aplurality of facets. As another example, data storage 1512 can includelight field representations from the dense field imager 1502 where eachlight field representation is provided by a facet in the dense fieldimager 1502. As another example, data storage 1512 can include a densefield representation resulting from combining light field data and/orraw image data. In yet another example, data storage 1512 can includecalibration information for the dense field imager 1502 that providesdepth and alignment information for pixels associated with imagingelements on facets. Data storage 1512 can include information thatassociates particular sets of image data with particular imaging devicesfrom the plurality of imaging devices 1504. Data storage 1512 caninclude image device alignment information that can be used ingenerating light field data. In some cases, data storage 1512 furtherincludes exposure balancing information to balance exposure levelsbetween the imaging devices 1504. Data storage 1512 can also includedistortion correction information, calibration information, pixelcorrespondence information, and the like.

In some embodiments, data storage 1512 includes an image data setcomprising a tensor derived from a first light field data setrepresenting a first portion of a light field, a second light field dataset representing a second portion of a light field, and informationrelating to a spatial relationship between the first and second portionsof the light field. The first and second portions of the light field canbe received directly from an imaging device (e.g., a dense field imager,a light-field camera, etc.), the light field data sets can be stored indata storage 1512, or the light field data sets can be received fromexternal data sources 1521. The light field data sets can be functionsof spatial coordinates, direction, wavelength, and/or time, and in someembodiments, the light field data sets can be represented using at leastfour dimensions. The light field data can be derived from pixel dataacquired by imaging elements in a dense field imager. An array ofimaging elements can make up a facet, and the pixel data acquired by thearray of imaging elements of a facet can be combined into light fielddata. Accordingly, the first light field data can be derived from pixeldata of a first facet and the second light field data can be derivedfrom pixel data of a second facet. The first and second facets can becoplanar or non-coplanar and they can have fields of view thatsubstantially overlap or that are substantially adjacent, with a portionof their fields of view overlapping.

The image processing system 1500 includes the pre-processing module 1514which is configured to receive data from the image data sources 1501 andprepare the data for conversion into light field data, dense field data,viewpoint images, or any combination of these. The pre-processing module1514 can use one or more computer processors to prepare the data.Preparing the data can include, for example, calibrating the receiveddata based on calibration data associated with the device that acquiredthe data. Calibration can include associating spatial and/or directionalcoordinates with pixels in the device acquiring the data. Using thiscorrespondence map, the pre-processing module 1514 can aligncorresponding portions of acquired images for later combination and/orconversion to light field representations or dense fieldrepresentations, as described herein. The pre-processing module 1514 canadjust received data based on properties of the sensors acquiring thedata, such as, for example, correcting for dynamic range differences,different sensitivities, different sizes, different resolutions, and thelike. In some embodiments, the pre-processing module 1514 receives rawimage data from a plurality of imaging elements and/or imaging devicesand creates separate images from the raw image data. In someembodiments, the separate images are aligned based at least in part oncalibration data and/or registration of overlapping images. In someembodiments, the aligned images are compared to a predicted model anddeviations from the model are assessed to identify outlier pixels and/ordeviations from the model. Based at least in part on these differences,the alignment parameters can be adjusted to improve alignment of pixeldata among imaging elements and/or imaging devices.

In some embodiments, the pre-processing module 1514 is configured tocombine pixel blocks from image sensors in an array of image sensors toextract improved color information. For image sensor arranged in a Bayerpattern such that a block of pixels are used to generate a single outputcolor (e.g., a 2×2 pixel pattern with a red pixel and a green pixel ontop and a green pixel and a blue pixel on bottom), the blocks of pixelsfrom multiple sensors can be combined to provide greater color databased at least in part on angular differences between a source and thepixel block capturing the incident light from the source. Because lightcolor corresponds to a frequency of light and different frequencies oflight behave differently through optical components when measured fromdifferent angles, an increase of color information can be extracted whencomparing pixel blocks on different sensors because they see the objectof interest from a different angle. This can improve processes such asedge detection and can reduce or eliminate halos from sharp transitionsin a real world image that happens to split a pixel block. This can alsolead to better output images, as they may be perceived as morerealistic. Thus, by comparing color differences between pixel blocksthat view an object from different angles, more accurate colorinformation can be extracted.

The image processing system 1500 includes the light field generatormodule 1516 which is configured to receive pixel data from a pluralityof imaging elements and to derive a light field representation based atleast in part on the received pixel data. Facets that include an arrayof imaging elements can be used to acquire data that is converted into alight field representation. The imaging elements can be coplanar andtheir optical axes can be primarily aligned along a direction. The lightfield generator module 1516 can receive this data from the facets of adense field imager (or after pre-processing with the pre-processingmodule 1514), for example, and generate a light field representation bydetermining a radiance of the light in the scene as a function ofposition and direction. In some embodiments, the light fieldrepresentation includes time information. In some embodiments, thespatial and directional components of the light field representation areexpressed using four dimensions, five dimensions, or more than fivedimensions. The light field generator module 1516 can be configured toreceive image data from a plurality of imaging devices 1504 and deriveat least one light field representation using the received image data.The light field generator module 1516 can also use calibrationinformation, alignment information, and the like to generate a lightfield representation. The light field generator module 1516 can beconfigured to receive image data from a variety of image sources and togenerate one or more light field representations from the received imagedata.

The image processing system 1500 includes the dense field generatormodule 1518 which is configured to receive a plurality of light fieldrepresentations and to combine them into a dense field representation.The dense field generator module 1518 can use one or more processors toderive an image data set comprising a plurality of light fieldrepresentations and a tensor describing spatial relationships betweenthe plurality of light field representations. In some embodiments, thederived image data set is represented using five dimensions for thelight field (e.g., three spatial dimensions and two directionaldimensions) with an additional one, two, three, four, or more than fourdimensions for the tensor describing the spatial relationships. In someembodiments, the dense field generator 1518 is configured to receive aplurality of light field representations, each representation expressedin five dimensions, and to derive the spatial relationship tensor oncombined light field representations. For example, two light fieldrepresentations having five dimensions each can be combined into arepresentation having 10 dimensions. In some embodiments, the spatialrelationship tensor can be determined using a representation with lessthan 5 dimensions, greater than 5 dimensions, greater than 10dimensions, greater than 20 dimensions, greater than 40 dimensions,greater than 60 dimensions, greater than 80 dimensions, greater than 120dimensions, greater than 200 dimensions, or greater than 500 dimensions.The dense field generator module 1518 can determine the spatialrelationship tensor on the 10-dimensional representation.

Once the spatial relationship tensor is determined, the dimensionalityof the combined light field representations can be reduced back down tofive or fewer dimensions. The light field representation resulting fromthe process can be a representation that uses non-Euclidean geometry, orthat is in a coordinate system that is not flat, but is curved or hasnon-planar or non-flat dimensions. The resulting spatial relationshiptensor can include one dimension, two dimensions, three dimensions, ormore than three dimensions. The resulting spatial relationship tensorcan describe mappings or transformations of light field representationsfrom the non-Euclidean geometry or the geometry of the combined lightfield representation into a Euclidean geometry or a traditionalgeometry, allowing the viewpoint generator module 1520 to extractviewable images from the combined light field representation. Thus, the10-dimensional combined light field representation can be reduced tofewer dimensions when represented as a dense field representation.

This process can be implemented on more than two light fieldrepresentations. This process can be used to combine light field datawhile maintaining the dimensionality of the light field representations,which can mean that combining a plurality of light field representationsresults in an intermediate light field representation having a number ofdimensions that is equal to the sum of the dimensions of the input lightfield representations. For example, if four 5-D light fieldrepresentations were combined, the intermediate light fieldrepresentation would have 20 dimensions. Combining the input light fieldrepresentations using this process can be accomplished wherein, duringthe combining process, a maximum total number of dimensions used is lessthan 5 dimensions, greater than or equal to 5 dimensions, greater thanor equal to 8 dimensions, greater than or equal to 10 dimensions,greater than or equal to 20 dimensions, greater than or equal to 40dimensions, greater than or equal to 60 dimensions, or greater than orequal to 80 dimensions. The process can be used to derive a data setthat reduces the number of dimensions of the intermediate representationto one that has the same dimensionality as any of the input light fieldrepresentations. In addition to the combined light field representation,the spatial relationship tensor can have an additional one, two, three,or more than three dimensions for the spatial relationship tensor. As aresult, a plurality of multi-dimensional light field representations canbe combined to form a dense field representation having the samedimensionality as the input light field representations with one or moreadditional dimensions to describe the spatial relationships of the inputlight field representations.

The image processing system 1500 includes the viewpoint generator module1520 which is configured to produce images and/or video using receivedimage data, light field representations, and/or dense fieldrepresentations. The viewpoint generator module 1520 can extractviewable images and/or video from a dense field representation using thespatial relationship tensor and the associated light field data. Thespatial relationship tensor can be used to define extraction parametersso that a viewable image can be extracted from the combined light fieldrepresentations. In a traditional light field representation, a 2D imagecan be extracted from the representation by effectively intersecting aplane at a desired location to determine radiance at each point on theplane, thus producing an image. Based at least in part on thenon-Euclidean and/or non-linear nature of the combined light fieldrepresentations, this process differs from the traditional process asthe required geometry of intersection is not necessarily a plane. Thespatial relationship tensor is used to define the correct operation toperform on the combined light field representation to extract theradiance at the desired points. The spatial relationship tensor can beused to extract 2D images, 3D images, stereoscopic images, images frommultiple viewpoints, video, holograms, and the like from the combinedlight field representations. Described in another way, the dense fieldrepresentation includes combined light field representations along witha spatial relationship tensor which can be used by the viewpointgenerator module 1520 to extract viewable images. Thus, the image dataset generated by the dense field generator module 1518 contains theradiance information for points within the scene as well as theparameters used to extract viewable images from the image data set. Insome embodiments, the viewpoint generator module 1520 can generateimages using super-resolution techniques, thereby generating an outputimage with an increased resolution over any of the input images. In someembodiments, the viewpoint generator module 1520 can generate viewpointsthat effectively see around and/or through occlusions by combining datafrom multiple viewpoints. In some embodiments, the viewpoint generatormodule 1520 can generate stereoscopic images and can vary aninter-ocular distance. The viewpoint generator module 1520 can beconfigured to change a point of view or a viewing angle of an outputimage, change a depth of field, change a focus depth, and to increase adynamic range of an output image compared to individual images receivedfrom the image data sources 1501.

The image processing system 1500 as described can provide very powerfulcreative flexibility after image acquisition (e.g., in post-processing).Thus, a post-processing system 1522 can be configured to receive imagedata, light field data, and/or dense field data from the imageprocessing system 1500. In some embodiments, the post-processing system1522 is separate from the image processing system 1500, and in someembodiments, the post-processing system 1522 is a part of the imageprocessing system 1500. The creative flexibility provided by thepost-processing system 1522 is based at least in part on the ability tomanipulate light field data and/or dense field data to accomplish anumber of post-processing tasks including, for example and withoutlimitation, re-focusing, changing a depth of focus, relighting based ondepth, changing a viewpoint, changing an inter-ocular distance forstereoscopic images, and the like. As one example, the user can re-focusin generally any desired manner using the post-processing system 1522.It can therefore be useful to provide some context regarding the intentof the cinematographer with regard to focus and other parameters whenthey shot the video. In this regard, metadata associated with a videofile can be used to store certain cinematographic settings defined bythe cinematographer or other camera operator during the shoot (e.g.,focus region or quality, depth-of-field, 3D convergence, inter-oculardistance, white balance, etc.). The metadata can then be used by thepost-processing system 1522 to access the user-defined settings. Forinstance, when the file is opened for playback, the post-processingsystem 1522 or other playback component may open the file according tothe stored metadata, with the initial focus quality or depth-of-fielddefined by the cinematographer, for example. The metadata can be storedin data storage 1512, for example. The post-processing system 1522 canbe configured to allow dynamic editing of a video, reducing a need fordetailed plans for capturing the video. For example, the post-processingmodule can change a focus, zoom in or out, change a viewing angle, andthe like to mimic similar effects traditionally performed by cameraoperators. Thus, the post-processing system 1522, using the dense fielddata, can change a focus of the creative process from acquiring thevideo in a particular way to editing the acquired video to achieve adesired effect. In some embodiments, the post-processing system 1522 canbe provided as a tool to a community of users. Raw dense field data canthen be provided to the community which could allow users to generatenew and unique videos based off the originally acquired video. Forexample, a movie can be shot and a particular version of the movie canbe generated for viewing by the general public. The raw data for themovie could be provided to the community of users to allow them tomodify aspects of the movie to generate a personalized version of scenesfrom the movie. A different character can be followed through a scene, afocus of the scene can change, a viewing angle can change, or anycombination of these to alter the users' experience. In someembodiments, the post-processing system 1522 can allow for dynamicfocusing and tracking of objects within a scene. For example, at sportsevent in a stadium, the ball and the players can be tracked dynamicallyso that each can be the focus of the output video stream. This can allowfor dynamically changing the focus depending on circumstances to, forexample, show what a critical player was doing at a particular pointduring the event. In some embodiments, the post-processing system 1522can enable a user to generate viewpoints from a large number of virtualcameras. This can reduce or eliminate the importance of placement ofactual cameras and/or the importance of camera operators.

The image processing system 1500 may also be configured to compressimage data for storage in data storage 1512. For instance, image datacorresponding to each imaging device in the array 1502 may be compressedseparately, and the compressed image data may be associated with thecorresponding imaging device in data storage 1512. Image datacompression and associated techniques are discussed in further detailbelow. In certain embodiments, the dense field imager 1502 stores“compressed raw” image data.

The image processing system 1500 can include a laptop, workstation, orother appropriate computing device configured to execute the variousmodules 1514, 1516, 1518, and 1520, which can include software modulesor programs. The image processing system 1500 is in communication withthe image data sources 1501 through any suitable connection (e.g., viaEthernet, USB, Wi-Fi, Bluetooth, WiGig, or some other appropriateconnection).

The post-processing system 1522 can ingest image data and metadata fromthe image processing system 1500. The post-processing system 1522 canthen use the metadata to make creative adjustments to a variety ofcinematographic parameters. Such parameters can include focus point orfocus quality, depth of field, inter-ocular distance, 3D convergence,and the like. Adjustment of particular aspects will be discussed ingreater detail below. The image processing system 1500 and/or thepost-processing system 1522 may additionally provide a graphical userinterface 1524 on a display 1523 coupled to the image processing system1500, for example.

Because the image data includes image data from a plurality of imagingelements and/or facets in the dense field imager 1502, or from eachimaging device in the array 1504, or from a plurality of light fieldsources 1506 or external data sources 1521, as well as correspondingmetadata, video files can include a tremendous amount of image data.Thus, the image processing system 1500 and/or the post-processing system1522 can additionally compress the received image data and/or metadata.In some cases, the image data and/or metadata is compressed according toa compression ratio of from between about 2:1 and about 30:1. In certainembodiments, the image data and/or metadata is compressed according to acompression ratio of from between about 2.5:1 and about 10:1. In oneembodiment, the compression ratio is about 3:1. In other cases, thecompression ratio can be at least about 2:1, at least about 4:1, atleast about 8:1, at least about 16:1, at least about 20:1, at leastabout 25:1, or at least about 30:1.

In other embodiments, the image processing system 1500 can be configuredto perform post-processing tasks. In some embodiments, the dense fieldimager 1502 can be configured to perform any or all of the functionsdescribed with respect to the image processing system 1500 and/or thepost-processing system 1522.

As described above, in certain embodiments, the imaging devices 1504 areseparate from one another. The array of separate imaging devices 1504can be oriented to capture a viewing theater, as described herein withreference to FIGS. 14A-C. The image processing system 1500 can beconfigured to process data from the imaging devices 1504. Each of theimaging devices 1504 may be a single-lens/single-sensor camera, or canhave multiple lenses and/or sensors. For instance, each of the imagingdevices 1504 could be one of the integrated multi-array imaging devicesdescribed herein. In any case, the imaging devices 1504 can includememory and an image processing module similar to the correspondingcomponents described with respect to the dense field imager 1502 havingthe integrated array of imaging elements arranged in facets, examples ofwhich are shown in FIGS. 1A-B, 5A-B, and 8-13.

Image Acquisition

FIG. 16 illustrates a flow chart of an example method 1600 for acquiringimage data with a dense field imager or an array of imaging deviceshaving overlapping fields of view wherein a plurality of light fieldrepresentations are generated using two or more coplanar imagingelements or imaging devices. Using the acquisition method 1600, viewableimages are extractable wherein the spatial and/or temporal resolution ofthe extracted images and/or video is substantially independent of thespatial and/or temporal resolution of the input image data. Thus, usinga plurality of imaging devices or imaging elements that acquire imagedata at individual acquisition resolutions and an acquisition framerates, video data can be extracted that has a resolution that is greaterthan any of the acquisition resolutions and a frame rate that is greaterthan any of the acquisition frame rates. The method 1600 receives rawimage data and generates dense field data through a process thatincludes pre-processing image data, combining image data, and generatingspatial relationship information about the combined image data. For easeof description, the method 1600 will be described as being performed byan image acquisition system. However, any step or combination of stepsin the method can be performed by the image processing system 1500 or adense field imager. Furthermore, the image acquisition system, the imageprocessing system 1500, and the dense field imager can perform anyindividual step or combination of steps in conjunction with the othersteps being performed on other systems. Thus, the description thatfollows is not to be interpreted as limiting the performance of theacquisition method to a single system or to a system that is separatefrom the dense field imager or the image processing system.

In block 1605, the image acquisition system receives raw image data froma plurality of imaging elements and/or imaging devices. The raw imagedata can be received directly from the imaging elements, or it can bereceived from a camera or dense field imager. Receiving raw image datacan include receiving an array of digitized values corresponding toquantities of light incident on a pixel of an image sensor. The rawimage data can include digitized data corresponding to color bands. Forexample, raw image data can include an array of digital values arrangedaccording to a known color pattern, e.g., a Bayer pattern. In someembodiments, monochromatic sensors can be used such that two, three,four, or more arrays of digitized values are received where each arraycorresponds to digitized values for a particular wavelength band. Forexample, a first array can correspond to a red sensor, a second arraycan correspond to a green sensor, a third array can correspond to a bluesensor, and a fourth array can correspond to a panchromatic or whitesensor. The image acquisition system can combine color information forgroups of pixels into a representative color at a location in thesensor. For example, to determine luminance, the image acquisitionsystem can de-Bayer the raw image data. As another example, usingmonochromatic sensors can allow the image acquisition system to filterout crosstalk between color pixels, thus improving the luminancedetermination. The image acquisition system can apply any calibrationinformation to the received raw data. For example, baseline values canbe corrected for (e.g., “dark” current or noise), responsecharacteristics can be corrected for (e.g., differing sensitivities),and the like.

In block 1610, the image acquisition system generates separate imagesfrom the received raw data. The image acquisition system can groupcalibrated raw image data into 2D bitmaps. When the raw data is providedby a dense field imager as described herein, the separate images fromimaging elements on a particular facet will be substantially similar asthey are pointed in generally the same direction and cover a similarportion of the scene being imaged. Separate images from other facets maybe substantially similar (e.g., where the facets are coplanar), they maysubstantially overlap with images from adjacent facet (e.g., where theoptical axes of the facets converge), or they may be substantiallyadjacent with images from adjacent facet (e.g., where the optical axesof the facets diverge).

In block 1615, the image acquisition system registers the separateimages. Registering the separate images can be important for latermapping pixel data to real world coordinates. One of the goals ofregistering the separate images, especially for images from the samefacet, is to determine relationships between pixels in the facet. Whenthe image acquisition system registers the images, it can create amatrix of correspondence vectors. The correspondence vectors can bemappings from one pixel to corresponding pixels in other sensors on thefacet. For example, for a first pixel on a first image sensor within afacet, a correspondence vector can be created that maps the first pixelto a corresponding pixel on the other sensors on the facet. Thecorrespondence vectors can be created for pixels that acquire image datathat is also acquired in other sensors. Thus, for “edge” pixels, orpixels that acquire image data that is not acquired by other sensors,there are no correspondence vectors. The correspondence vectors can berepresented as a magnitude and an angle that maps a first pixel to alocation of a corresponding pixel. For instance, for a facet with Nimaging elements, each pixel in a sensor can have about N−1correspondence vectors associated therewith, corresponding to vectorsfrom the pixel's position to a position of a corresponding pixel inanother imaging element in the facet. Registration can be performedusing intensity-based and/or feature-based techniques.

In block 1620, the image acquisition system compares the registeredimages to predicted results and aligns the registered images. Comparingthe registered images to predicted results can include modeling behaviorof the imaging element, such as the optics, and filtering out expectedsignal from noise. This procedure can improve registration as itimproves signals which in turn can improve registration based at leastin part on being able to more closely match real values of pixels whenperforming intensity-based registration, for example. In someembodiments, the optics on the facets or other imaging devices makesthem epipolar. To account for this, the image acquisition system candetermine a curvature associated with the system and identify deviationsfrom the expected for each pixel. Outlier pixels can be removed from theimage data. The output of this step is a conformed set of correspondencevectors, where the conformation vectors represent corrections to thecorrespondence vectors based on models.

The image acquisition system can align the registered images to assignweights to angular data associated with light field representations.Based at least in part on the registered images, pixel data can beassociated with real world coordinates. By combining the separate imagesand comparing the real world coordinates, angular luminance informationcan be extracted for positions within the imaged scene. By comparing theluminance of different pixels, weights can be assigned to the conformedcorrespondence vectors according to warping, where the angle correspondsto the inverse of the warping. In some embodiments, it may beadvantageous to de-Bayer on a pixel by pixel basis as four times theangular information is available by treating each filtered pixel as aseparate pixel value rather than combining all filtered pixels within ablock to determine a luminance. From the weighted conformedcorrespondence vectors, five dimensional light field data can begenerated. In some embodiments, the five dimensions can be reduced tofour dimensions through a choice of parameterization, as describedherein.

In block 1625, the image acquisition system fuses the registered andaligned images. Fusing the registered and aligned images can includecombining light field data on a facet-by-facet basis. The imageacquisition system can take the light field data for a given facet alongwith the weighted conformed correspondence vectors to generate afive-dimensional tensor field. The five-dimensional tensor fieldincludes information about all the light field representations generatedfor each facet. In some embodiments, data can be combined in an additivemanner where light field data is aligned. In some embodiments, data isaveraged over position and/or direction to fuse the light field data. Insome embodiments, the image acquisition system can generate a singlemulti-dimensional data structure representing the image data for all thetiles in the dense field imager. The number of dimensions can correspondto the dimensionality of the input light field representations. Forexample, if five dimensions are used for the light field representationsgenerated in block 1620, then the fused tensor will have five dimensionsas well as it is a combination of the light field representations of thefacets.

In block 1630, the image acquisition system stitches the fused images.Stitching can include projecting the fused tensor into a universalimaging space where the universal imaging space can be a space where alllight field rays are parallel. The image acquisition system can thendetermine corresponding patches from one facet to adjacent facets. Theimage acquisition can then blend the patches to generate a single lightfield representation across the universal imaging space. The imageacquisition system can extract depth information and flow fieldinformation from the fused tensor to enable stitching of the light fieldrepresentations from adjacent facets.

In block 1635, the image acquisition system removes redundant data fromthe combined image data. After fusing and stitching the light fieldrepresentations, the image acquisition system can identify and removelight rays that provide redundant information. As an example, afterfusion and stitching, the combined light field representation caninclude millions of rays that provide substantially similar information.The image acquisition system can sample these rays to reduce the numberof rays such that sufficient information is maintained to extractviewable images at targeted or desired resolutions and quality. Theimage acquisition system can be configured to determine which raysdeliver substantially the same information and to collapse these raysinto a single light ray. In some embodiments, this process can removeone or more dimensions from the combined light field representation.

In block 1640, the image acquisition system projects the combined imagedata into a multi-dimensional tensor representation. The imageacquisition system can reduce the dimensionality of the combined lightfield representation by projecting it into a space having a target ordesired number of dimensions. For example, the combined light fieldrepresentation can be projected into a geometry or space where fivedimensions are used to represent the light field data. As a result ofthis projection, an additional spatial relationship tensor is createdwhich maps the resulting multi-dimensional light field representationfrom a first geometry which can be non-Euclidean, non-linear, curved, orotherwise irregular, into a Euclidean or linear geometry. The spatialrelationship tensor can be one dimension, two dimensions, threedimensions, four dimensions, or more than four dimensions. The spatialrelationship tensor can be used to extract viewable images from themulti-dimensional light field representation. A dense field data set ordense field representation can comprise the combination of themulti-dimensional light field representation with the spatialrelationship tensor. Thus, dense field data can be a multi-dimensionaltensor representation. This multi-dimensional tensor representation canthen be stored on any suitable computer-readable memory device block1645, as described elsewhere herein.

Calibration

FIG. 17 illustrates a flow chart of an example calibration method 1700for determining corrective factors for imaging elements on a dense fieldimager or an array of imaging devices. The method generally uses aimages of a well-known object to determine correspondences betweenpixels of different imaging elements or imaging devices. To extractposition and direction information when generating light field dataand/or dense field data, calibration information determine using themethod 1700 can be used to correlate 2D pixel coordinates to 3D imagescene coordinates. The calibration method 1700 can be used to derivecalibration information for pixels within an imaging element and/orwithin an imaging device, which calibration information can be usedduring image processing to generate quality light field representationsand dense field representations. For ease of description, the method1700 will be described as being performed by a calibration system. Thecalibration system can be a part of a dense field imager, an array ofimaging devices, an image processing system, a module on a computingdevice, or it can be distributed across a plurality of these and othersystems and/or devices. One or more steps in this method can beperformed by any one or any combination of elements or modules in thedevices and systems described herein.

In block 1705, the calibration receives images of a well-known object inspace acquired with a dense field imager. It is to be understood, thatalthough much of the description is presented as being performed withimages acquired with the dense field imager, any array of imagingelements and/or imaging devices may be used. The well-known object inspace can be a calibration target. An example calibration target 1800 isillustrated in FIG. 18A. The example calibration target 1800 is aninternal corner of a cube with a plurality of circular control points1802 placed at known locations on the calibration target 1800. Thecalibration target 1800 can be any suitable target that provides aplurality of control points at known locations. Some calibration targetscan include an interior of a sphere, ellipsoid, or polyhedron, orcontrol points placed at known locations without conforming to a regularshape. In some embodiments, the control points can be placed randomly ifthe locations are known or can be determined. Images of the calibrationtarget 1800 can be acquired with the dense field imager, wherein theimages include a plurality of images taken with the imaging elements ofthe dense field imager. In some embodiments, all of the imaging elementsacquire images of the calibration target 1800. In some embodiments, lessthan all of the imaging elements acquire images of the calibrationtarget 1800 and methods are used to interpolate calibration constantsfor imaging elements that did not acquire an image of the calibrationtarget 1800.

In block 1710, the calibration system extracts circular control points1802 from the received images. The calibration system can use sub-pixelimage processing to extract this information. In some embodiments, thecontrol points 1802 are visually distinguishable from the calibrationtarget by being a different color, a different brightness, having adifferent texture or pattern, or a different spectral signature when animage is acquired. The calibration system can be configured to identifythe control points based at least in part on the differentcharacteristics of the control points 1802 on the calibration target1800. In some embodiments, the calibration system begins with apredicted location for the calibration targets 1802 and iterativelycorrects the predicted location until it finds an actual location of acontrol point 1802. The difference between the predicted location andthe actual location within the image can then be used, at least in part,to derive calibration information. In some embodiments, the calibrationsystem is configured to extract position information for the controlpoints 1802 based at least in part on the properties of the controlpoints 1802. For example, the control points 1802 can include a barcode, QR code, or other visual indication that can be extracted by animage processing system where the visual indication provides eitherabsolute or relative position information. Extracting control points1802 can include identifying pixels that include information about acontrol point and creating a table or other data structure that mapspixels to control points.

In block 1715, the calibration system associates a resulting set of 2Dcoordinates for each imaging element with a corresponding set of 3Dcoordinates derived from the physical dimensions of the calibrationtarget. The calibration system can be configured to map a correspondencebetween a centroid of a control point to a centroid of the pixels thatacquired image data of the control point. Based at least in part on thecorrespondences and the known locations of the pixels and controlpoints, the calibration system generates a map between a pixel locationand a corresponding location in 3D space in a scene being imaged.

In block 1720, the calibration system determines internal and externalgeometrical camera parameters from the set of 2D-to-3D correspondences.The internal geometrical camera parameters can include distortioncoefficients for pixels within an imaging element and the externalgeometrical camera parameters can include relative orientationinformation for every imaging element in the dense field imager.

Due in part to the imaging elements acquiring images of the calibrationtarget from a variety of angles, there is a distortion in the images ofthe calibration target 1800 that can be corrected using thecorrespondence maps. Using internal camera parameters determined usingthe method 1700 or other similar method, images can be geometricallycorrected and undistorted. For example, FIG. 18B illustrates an exampleplot 1830 showing results of corrected and non-corrected pixel data. Theoriginal positions of the control points without distortion removed showmisalignment artifacts introduced by optics and other effects. Near thecenter of the image, where the control points are relativelystraight-ahead of the imaging elements, there is little differencebetween the determined control point positions with and withoutdistortion corrections made. However, the control points at the edges ofthe images, the control points with the most extreme angles within theimages, have distorted positions. By applying the internal cameraparameter calibration values to remove distortions, the actual positionsof the control points can be recovered, as is shown by the plot 1830.

Imaging elements can have optical axes that are not parallel with oneanother and the correspondences can be used to determine relativeorientations of imaging elements. With the generated external cameraparameters, the position for every imaging element in the array relativeto the calibration target can be calculated. For example, FIG. 18Cillustrates a plot 1860 of relative imaging element orientationdetermined from the external camera parameters. The relative orientationof two imaging elements is indicated by the axes labeled 1862 a and 1862b. The relative orientation indicates the direction of the optical axesof the two imaging elements (the arrow pointing toward the calibrationtarget) with additional axes defining the horizontal and vertical forthe respective imaging elements.

One measure for a quality of calibration is a vertical misalignment ofcorresponding control point locations in a set of two imaging elementsafter calibration corrections are applied. After applying thecalibration method 1700 on a dense field imager, the plot 1890illustrated in FIG. 18D is achieved. The plot 1890 demonstrates that thevertical misalignment, on average, is approximately zero with a spreadin the distribution of less than about 0.1 pixels.

In block 1725, the calibration system can store the resultingcalibration parameters in data storage, such as the data storage 1512described with reference to FIG. 15. The calibration parameters can beretrieved and utilized by a dense field imager, an array of imagingdevices, an image processing system, a computing device, or anycombination of these to produce light field data, dense field data,image data, or the like.

Light Field Stitching

FIG. 19 illustrates a flow chart of an example method 1900 for stitchinglight field representations. Stitching light field representations caninclude combining light field representations from one or more lightfield data sources. The light field data sources can include storedlight field data, an array of plenoptic or light-field cameras, an arrayof imaging devices, a dense field imager, or any combination of these.Stitching light field representations can be used to expand a field ofview covered by light field data. For example, a first light fieldrepresentation can cover a first field of view and a second light fieldrepresentation can cover a second field of view, the second field ofview overlapping with the first field of view. The method 1900 can beused to combine the first and second light field representations toproduce a dense field representation that includes a combined lightfield representation and a spatial relationship tensor such that thecombined field of view of the resulting dense field representationcovers the first and second fields of view. For ease of description, themethod 1900 will be described as being performed by the image processingsystem, but any suitable device or combination of devices or systems canbe used to perform the method. For example, a dense field imager can beconfigured to stitch light field representations. Any one or anycombination of the following steps of the method 1900 can be performedby one or more components of the image processing system, the densefield imager, a computing device, and the like.

In block 1905, the image processing system acquires pixel data from aplurality of imaging elements. In some embodiments, the pixel data canbe acquired directly from the imaging elements on a facet of a densefield imager. In some embodiments, the pixel data is acquired and storedand then retrieved by the image processing system. The pixel data caninclude metadata or other configuration or calibration parameters thatfacilitate and/or improve processing of the pixel data. Alignment orregistration information of the pixel data can be included that maps the2D pixel data to 3D image scene data. In this way, pixel data can bemapped to spatial and directional coordinates which can facilitatefurther processing.

In block 1910, the image processing system generates a plurality oflight field representations. Generating a plurality of light fieldrepresentations can include grouping planar imaging elements to create alight field representation. In some embodiments, the planar imagingelements also have substantially parallel optical axes. The plurality oflight field representations can be generated based at least in part ongeometrical considerations. For example, a first light fieldrepresentation can be generated using pixel data acquired from imagingelements aligned generally in a first plane and a second light fieldrepresentation can be generated using pixel data acquired from imagingelements aligned generally in a second plane, the second planenon-coplanar with the first plane. In some embodiments, the plurality oflight field representations are generated based at least in part onlogical groupings. For example, a first set of imaging elements can becoupled to a first acquisition and/or processing electronics, such as ina facet of a dense field imager. A second set of imaging elements can becoupled to a second acquisition and/or processing electronics. The firstset of imaging elements and the second set of imaging elements can beused to generate first and second light field representations,respectively. The first and second sets of imaging elements can becoplanar or the first set can be non-coplanar with the second set. Insome embodiments, the fields of view of the generated light fieldrepresentations overlap. Generating a light field representation caninclude using a plurality of images to define a data set that describesradiance as a function of position and direction. In some embodiments,the generated light field representations can be functions of timeand/or wavelength. In some embodiments, the five dimensions representingspace and direction can be reduced to four dimensions using aparameterized space, such as using a light slab, light spheres, a planeand angle, and the like.

In block 1915, the image processing system creates a spatialrelationship tensor that defines spatial relationships among the lightfield representations. To combine light field representations, the imageprocessing system can join the light field representations into arepresentation of a higher dimensional order to accommodate thedimensionality of each of the component light field representations. Forexample, if the image processing system were to join two 5D light fieldrepresentations, the image processing system could create a 10Drepresentation accommodating the two 5D representations. The imageprocessing system can then process the combined light fieldrepresentation to identify spatial relationships between the light fieldrepresentations. As a result of this determination, a spatialrelationship tensor can be created that effectively maps light fieldrepresentations in relation to one another. The spatial relationshiptensor can be used, for example, to generate viewpoints from a resultingcombined light field representation. For example, for a single lightfield representation a 2D bitmap can be generated by bisecting the lightfield representation with a flat plane. The spatial relationship tensorfor this scenario would be used to identify the flat plane to use toextract the desired image. Similarly, for combined light fieldrepresentations, the geometry may not be flat or Euclidean and thespatial relationship tensor can be used to identify the correct “shape”(analogous to a plane to generate a 2D bitmap from a single light fieldrepresentation) to use to bisect the combined light field data toextract a desired viewpoint.

In some embodiments, stitching light field representations comprisesprojecting light field representations into a universal imaging spacewhere all rays are parallel. Light field data acquired with adjacentimaging devices or facets can then be joined by finding correspondingpatches between the light representations and blending the light fieldrepresentations at those corresponding patches. Finding thecorresponding patches can, at least in part, provide information fordetermining the spatial relationship tensor as the correspondences arerelated to the spatial relationships between the light representations.

In block 1920, the image processing system uses the spatial relationshiptensor to combine the light field representations to create a densefield representation. Once the spatial relationship tensor isdetermined, the combined light field representation can be projectedinto a reduced-dimension tensor, e.g., a 5D tensor. The combination ofthe 5D tensor representing the combined light field representations andthe spatial relationship tensor can comprise the dense fieldrepresentation. With this information, a viewpoint generator cangenerate viewpoints and a post-processing system or module can achievelight field-type effects. The light representations have beeneffectively joined to create a dense field representation over thecombined field of view.

An example of where this method may be applied is shown in FIGS. 20A and20B. FIG. 20A illustrates a scene to be imaged 2000. A region ofinterest 2002 is indicated with a rectangle and a field of view 2004 ofa device providing light field data is indicated with a second, dashedrectangle. The field of view 2004 of the light field device is shown inFIG. 20B. To cover the region of interest 2002, then, a plurality oflight field representations can be joined. The method 1900 can be usedto join the light field representations by intersecting the light raysof the light field representations to be joined. Intersecting the lightrays comprises matching vectors in 4 or more dimensions. By finding theintersection of the vectors, information can be determined that maps thelight field representations into a single light field representation ina coherent geometry from which any cross section will result in a validbitmap across the cumulative field of view. In some embodiments, thecross section is not a plane and is determined using the spatialrelationship tensor.

In block 1925, the image processing system stores the dense fieldrepresentation. In some embodiments, the dense field representation canbe stored in data storage 1512 which is described with reference to FIG.15. In some embodiments, the dense field representation can include theoutput combined light field representation, the spatial relationshiptensor, the acquired pixel data, the generated light fieldrepresentations, metadata, or any combination of these.

In some embodiments, combining the light field representations includesstitching together light field representations having substantiallynon-overlapping fields of view to create a dense field image set havinga substantially wider field of view than the individual light fieldrepresentations. The combined lateral field of view of the dense fieldimage set can be greater than or equal to about 145 degrees in someembodiments.

In some embodiments, the method 1900 of stitching light fieldrepresentations is independent of where imaging elements and/or imagingdevices are positioned relative to one another. This can be accomplishedby intersecting the light rays of the input light field representationsas that process will remove dependence, at least in part, on sensorposition. In some embodiments, relationships between any number of lightfield representations that share a sufficient field of view should becombinable into a single tensor lattice (e.g., a combined light fieldtensor in combination with a spatial relationship tensor) from whichimages can be created.

Depth Mapping and Focus and Depth of Field Control

A dense field imager and/or multiple imaging devices can be used toprovide depth mapping. Close objects will shift more in position betweenimaging devices than objects that are far away. A depth map can becreated using a determination of how much an object shifts betweenimages. In one embodiment, some imaging devices are used to create adepth map, while other imaging devices are used to capture images withdifferent focus settings. Depending on the embodiment, the depth mappingtechniques described herein may be used with any of the systemsdescribed herein having an array of physically separate cameras (e.g.,with respect to FIGS. 14A-C or FIG. 15) or, alternatively, to any of thecamera systems having an integrated array of imaging elements (e.g.,with respect to FIGS. 1A-B, 5A-B, and 8-13).

For example, some of the imaging devices could use different focaldistances, and the depth map would provide information about whichimaging devices would best represent an object in focus. As anotherexample, some of the imaging devices could use different apertures andthe depth map could be used to select the desired depth of field fordifferent portions of the resulting image. In yet another embodiment,the depth map could be used to artificially create or adjust focus,bokeh and/or depth of field during post-processing.

Where the imaging devices have a relatively wide depth of field (e.g.,at or near infinity), the native images captured from the imagingdevices may be substantially in-focus, across the entire depth of theimage scene. In such cases, the depth map can be used to createartificial bokeh, simulating out-of-focus quality in desired imageregions. For instance, the depth map can be used to simulate auser-selected region of focus or depth-of-field by blurring imageregions outside of the selected region of focus or depth of field.

In one embodiment, the user selects a focus region while shooting. Theuser may select a region of focus using the viewfinder touch screen,cursor, voice command, or other appropriate input mechanism, and thecamera system stores this desired focus region in metadata associatedwith the video file. The system may also add bokeh to the image that isdisplayed on the viewfinder in response to the user selection.

As discussed, in post-processing, the focus or depth of field set by thecinematographer during shooting may be initially presented to the user.But, depending on the desired creative effect, and regardless of thecinematographer's settings, the user can adjust the focus or depth offield as desired, providing substantial creative flexibility.

The depth map can also be used to control the 3-D effect of an image.The depth map may be shown in substantially real time. Industries suchas, for example, robotics, biometrics, security, surveillance andautonomous vehicles can advantageously utilize depth information.

The depth map can be created by the post-processing module. In anotherembodiment, the image processing module creates the depth map.

Moreover, while the transition from in-focus to blurry regions intraditional cameras may have a certain characteristic response dependingon the aperture of the lens, the synthetic blurring effect achievableusing the multi-imaging device cameras described herein can be tailoredto achieve generally any desired blurring effect. For instance, thetransition from in-focus to blurry regions, can be linear, logarithmic,exponential, step-function, or a combination thereof.

The multi-imaging device techniques can also be used to provideinteractive focus and depth of field during playback. For instance, theuser in some cases can select a first point of focus and/or depth offield to view an initial segment of a video clip, and then change thepoint of focus or depth of field as desired, during viewing. As oneexample, during instant replay of a catch in a football game, the videooperator may be able to adjust the depth of field depending on themovement of the football or of the receiver. Where the imagingprocessing system on the camera generates the combined video image dataon camera, the user may use the touch screen interface and/or controlson the camera to pause playback, modify the focus or depth of field, andthen resume playback, as desired.

In these and other ways, certain embodiments described herein cancapture full light field effect image data and/or provide similaradvantages to light-field and plenoptic cameras, such as the ability toadjust focus and/or depth of field in post-processing.

Three Dimensional Video with an Imaging Device Array

Images taken from two different perspectives can be combined in avariety of ways to create an illusion of depth. In one example, glassescontaining liquid crystal may block or pass light in synchronizationwith an alternating display of images, so that the left eye sees adifferent image than the right eye. In another example, glassescontaining linearly polarized lenses or circularly polarized lenses mayblock or pass light corresponding to the polarized projection of twodifferent images. Other examples that use glasses include, for example,Infitec (interference filter technology), Inficolor 3D, complementarycolor anaglyphs, ColorCode 3D, Chromadepth. Other examples, which do notrequire glasses, are referred to as autostereoscopy.

A stereoscopic image typically uses two or more pictures taken fromdifferent perspectives. For example, imaging devices 200 a and 200 b inFIG. 7A, or any different subset of one or more cameras 1402 in any ofthe FIGS. 14A-C, provide pictures from one pair of differentperspectives. Imaging devices 200 a and 200 c provide pictures fromanother pair of different perspectives. The illusion of depth changeswith the distance between the imaging devices, so different pairs ofimaging devices provide different perceptions of depth. While theexemplary illustration in FIG. 7A provides three imaging devices, it isto be understood that multiple imaging devices could be used, providingthe director with multiple perspective options.

The perceived depth of objects in an image is affected by the distanceand/or relative angle between perspectives. Multiple images with varyingdistances between the images, or inter-ocular distance, can be acquiredto provide options for post processing. Moreover, multiple images withvarying angles (e.g., 3D convergence values) between the images, can beacquired to provide options for post processing. As mentioned above, aninitial inter-ocular distance and/or 3D convergence values can be set bythe cinematographer during the shoot, and the post-processing softwarecan load this pre-set value as a default when the video file is loadedfor playback or editing. The user can then adopt the defaultinter-ocular distance or adjust the distance in post as desired.

In some cases, a first set of at least one sensor is displaced from asecond set of at least one sensor by a distance within the range of fromabout 1 inch to about 18 inches, often within the range of from about1.5 inches to 6 inches within the same single hand held device, tocapture stereoscopic or full light field effect image data. In othercases, separate cameras located at spaced apart locations in a viewingtheater are used.

For cases where there is a need to accurately represent motion, it ispreferred that the multiple images are captured at the same time.

High Resolution Video with an Imaging Device Array

As discussed, an image processing system can combine or stitch togetherthe image data from the individual imaging devices. And as mentioned,with respect to FIG. 7A and FIGS. 19 and 20A, for example, the stitchingcan result in an effective resolution of the combined image that issignificantly higher than the individual sensors.

Another technique that can be used to improve resolution involvesgenerating multiple sequential image samples at slightly offset spatialpositions and/or different incident angles. For example, a mechanicallyactuatable optical element can be positioned before an array of imagingdevices, such as in the array of imaging elements illustrated in any ofFIGS. 8-13.

The optical element can be configured to wobble or otherwise move todirect light rays onto the array at slightly different spatial positionsand/or at incident angles. The optical element can include a transparentplanar member, and can be formed of glass, plastic, or some otherappropriate optically transmissive material. The optical element caninclude a refractive lens in some cases. In some embodiments, amechanical actuator at each corner of the optical element can beincluded, although any appropriate mechanism can be used. Movement ofthe optical element can result in a corresponding angular variation inthe light exiting the optical element. The effect of this variation isthat, depending on the current position of the optical element, thelight rays are directed towards slightly different portions of the arrayand at different angles of incidence.

In some embodiments, the optical element is actuated to direct the lightrays in a repeating pattern, which can vary, depending on theembodiment. For instance, a square pattern can be used, in which theoptical element directs the rays one or more pixels (or a fraction of apixel) right, then down, then left, and then up return to the originalposition. Or the light rays can be moved in a circular pattern. Theimaging devices can be configured to sample the image at differentpoints along the pattern trajectory.

When the optical element is transitioned through the pattern rapidly,each individual pixel in the imaging devices captures a correspondingpoint in an image scene in relatively rapid succession, but at slightlyoffset spatial positions and differing incident angles. The resultingsequentially captured images can then be combined to create higherresolution image frames. For instance, the image processing system cancombine the successively captured images to create high resolution videoframes.

Because the samples are captured sequentially, there can be some amountof motion in the image scene that occurs between successive samples.This can result in blur in the combined image. In some embodiments, thecamera includes a control that allows the user to adjusting the samplingrate to achieve a desired level of blur. In other cases, such as whereany amount of blur is undesirable or where lower resolutions areacceptable, movement of the optical element can be disabled. Forinstance, the camera can include a button, switch, touch screen controlor other appropriate mechanism to disable movement of the opticalelement. In yet other configurations, the optical element is notincluded.

The camera system can be configured to output video at a variety ofresolution levels, and the output resolution levels can be independentof the input resolution levels. For instance, any of the imaging systemsdescribed herein can generate image data having “5 k” (e.g., 5120×2700),Quad HD (e.g., 3840×2160 pixels), “4.5 k” resolution (e.g., 4,520×2540),“4 k” (e.g., 4,096×2,540 pixels), “2 k” (e.g., 2048×1152 pixels) orother resolutions. As used herein, in the terms expressed in the formatof xk (such as 2 k and 4 k noted above), the “x” quantity refers to theapproximate horizontal resolution. As such, “4 k” resolution correspondsto about 4000 or more horizontal pixels and “2 k” corresponds to about2000 or more pixels. The dense field imager, for example, can acquiredata that a viewpoint generator can use to output video image datahaving resolutions of at least 2 k, at least 4 k, at least 4.5 k, atleast Quad HD, at least 5 k, at least 6 k, at least 10 k, or at least 18k, at least 20 k, or greater.

In certain embodiments, the imaging systems described herein can alsoachieve relatively high zoom ratios. For instance, in one embodiment,the dense field imager can produce output data that can achieve a zoomration of 100:1. In various embodiments, at least about 10:1, at leastabout 25:1, at least about 50:1 and at least about 100:1 zoom ratios arepossible. In addition, the imaging elements of the dense field imagercan sample image data at relatively high frame rates. For instance, inone embodiment, the imaging elements can capture image data at a framerate of at least 500 frames per second. In various embodiments, theimaging elements can capture image data at a frame rate of at least 150frames per second, at least 200 frames per second, at least 300 framesper second, at least 400 frames per second, or at least 500 frames persecond, and at a resolution of at least 2 k. In other embodiments, thecamera 502 system can capture image data at a frame rate of at least 150frames per second, at least 200 frames per second, at least 300 framesper second, at least 400 frames per second, or at least 500 frames persecond, and at a resolution of at least 4 k.

High Dynamic Range Video Using an Imaging Device Array

The imaging devices described herein, for example, could be configuredto capture images using different exposure times. The images could thenbe combined to create a high-dynamic range image. Motion may affect thequality of images captured using different exposure times.

The imaging devices could also be configured to apply different gains tothe signals coming from the sensors. This is comparable to adjusting theISO setting on traditional cameras. Increasing the gain allowsmeasurement in low-light situations, with the trade-off of increasednoise in the measurement. The image could then be reconstructed usinglow-gain imaging devices for areas of the image that have sufficientlight, and high-gain imaging devices for areas of the image that need aboost in the signal.

A combination of different exposure times and different gain settingsfor the various imaging devices could also be used. Additionalcompatible high dynamic range techniques are discussed below and aredescribed in U.S. Patent Application Publication No. 2012/0044381entitled “HIGH DYNAMIC RANGE VIDEO”, which is incorporated by referenceherein in its entirety and is included in the attached Appendix.

High-Speed Videography

FIG. 21 illustrates a timing diagram for capturing images using multipleimaging devices. The timing of the image captures may be arranged toincrease the frame capture rate for video. For example, where a singleimaging device might be capable of capturing 60 image frames per second,three imaging devices could be used to capture 180 image frames persecond. In order to reduce the effects of “rolling shutter,” each framecould comprise rows obtained from multiple imaging devices. For example,the first third of the frame might come from a first imaging device, thesecond third of the frame might come from the second imaging device, andthe last third of the frame might come from the third imaging device. Asillustrated in the timing diagram shown in FIG. 21, the first third ofthe rows of a first sensor, the second third of the rows of a secondsensor, and the last third of the rows of a third sensor are captured atapproximately the same time. While three sensors were used forillustration purposes, it is to be understood that other numbers ofsensors may be used.

Other Image Processing Operations

The image processing systems described herein, such as the imagingprocessing system 102 of FIG. 1A, the image processing system 1500 ofFIG. 15, or the dense field imagers of FIGS. 8-13 can perform a varietyof other operations on the image data. For example, the image processingsystem may optionally compress the data from the imaging devices,perform pre-compression data-preparation (e.g., pre-emphasis and/orentropy reduction), format the compressed data, and the like. Examplesof such techniques are described in greater detail in U.S. Pat. No.7,830,967 entitled “VIDEO CAMERA” (the '967 patent), which isincorporated by reference herein in its entirety. In some instances, theimage processing system processes the image data to tune the imagestreams to perceptual space. Additional compatible techniques includegreen data modification and pre-emphasis processes shown and describedthroughout the '967 patent (e.g., with respect to FIGS. 8-11 and columns11-13). In general, certain embodiments described herein are compatiblewith and/or are components of embodiments described in the '967 patent.Thus, some or all of the features described herein can be used orotherwise combined with various features described in the '967 patent,including features described in the '967 patent which are not explicitlydiscussed herein. The '967 patent is included in the attached Appendix.

Viewpoint Generation

FIG. 22 illustrates a flow chart of an example method 2200 forgenerating a viewpoint using dense field data. The method 2200 can beused to extract viewable images and/or video in a variety of formats andresolutions. The dense field data allows for the spatial and/or temporaloutput resolution to generally be independent of the input spatial andtemporal resolutions. For ease of description, the method 2200 will bedescribed as being performed by the viewpoint generator, but it is to beunderstood that any suitable image processing system, post-processingsystem, dense field imager, computing device, or the like can be used toperform any of the following steps and any combination of devices can beused to perform the method 2200.

In block 2205, the viewpoint generator receives a dense fieldrepresentation. The dense field representation can be provided by, forexample, an array of imaging devices, a dense field imager, an imageprocessing system, data storage devices, a computing device, or anycombination of these. The dense field representation can include lightfield data from a plurality of devices that cover a combined field ofview. Light field data can be combined, as described herein withreference to FIG. 19, to generate dense field data. The dense field datacan include light field information and spatial relationshipinformation. The spatial relationship information can be used toselectively bisect a portion of the combined light field representationto extract a viewable image. The dense field data can providepost-processing capabilities similar to light field data, such asre-focusing capabilities, the ability to generate a variety of viewingangles, change inter-ocular distance for 3D images, and the like.

In block 2210, the viewpoint generator determines a portion of the densefield representation that contains the region of interest. Because thedense field data can cover a relatively large field of view (e.g.,potentially 360 degrees in azimuth and 180 degrees in elevation), theviewpoint generator determines a region of interest within the scenedata. The viewpoint generator then uses the spatial relationship matrixto determine a size and shape of an object to use to bisect the combinedlight field data to extract a viewable image of the region of interest.In some embodiments, determining the portion of the dense field datathat contains the region of interest includes extracting one or moreimages from the dense field data and refining the bisection object tofind the desired view of the region of interest.

In block 2215, the viewpoint generator calculates an optical flow and/ormotion compensation related to the region of interest. The optical flowcan include image artifacts related to positional and/or motion effectsof the imaging elements used to acquire the image data and/or theobjects being captured. The optical flow can be used to correct foraberrations that could potentially adversely affect an output image orvideo. Motion compensation can account for differing positions ofobjects when taken at different points in time. In some embodiments, theviewpoint generator can interpolate and produce images at viewpointsand/or times where no image data was actually acquired. For instance,image data from imaging elements positioned at a variety of viewpointscan be combined to create a virtual viewpoint. As another example, imagedata taken at different points in time can be used to generate a view ofthe region of interest at a time when no data was actually acquired.

In block 2220, the viewpoint generator digitally re-focuses on theregion of interest. The viewpoint generator can manipulate the densefield data to produce an image that is digitally focused on an object ofinterest. The viewpoint generator can change a focus depth and/or adepth of focus. In this way, different objects can be in or out of focusand a greater or smaller percentage of the generated viewpoint can befocused.

In block 2225, the viewpoint generator renders an image or video of theregion of interest. Using the optical flow, motion compensation, regionof interest, and focus information derived in the preceding steps, theviewpoint generator can extract viewable images or video from the densefield generator. The output resolution can be independent of theresolution of the imaging devices used to acquire the image data.

In this way, the viewpoint generator can be configured to synthesizeimages from any dense field data set (e.g., data acquired using multiplesensors, multiple cameras, array of cameras, etc.). The viewpointgenerator can be configured to generate pixel data, to data for blocksof pixels, shapes data, and finally images. The viewpoint generator canprovide a single integrated view from disparate sources. For example,data from disparate imaging devices can be combined into a dense fielddata set and the viewpoint generator can extract a single, integratedview of a scene from the disparate imaging devices. In some embodiments,the viewpoint generator can be configured to extract data at a varietyof dimensions such as, for example, 1D pixels, 2D bitmaps, 3D temporalimage sequences, 4D stereoscopic sequences, 5D light fieldrepresentations (for holograms), etc. In some embodiments, the viewpointgenerator can be configured to produce automated compositions based oninput criteria. For example, the viewpoint generator can be configuredto focus on an object of interest when a particular object is within adefined space. In some embodiments, the viewpoint generator can becoupled to a networked input source that receives information frommultiple users over a social network to provide dynamic viewpointsaccording to user requests.

Manufacture of Imaging Elements

In some embodiments, one or more liquid lens cells may be used with asmall sensor in an imaging device. The imaging device may then beattached to an adjustable frame.

Some optics may be implemented in the dielectric stack of the integratedcircuit using refractive microlenses or diffractive gratings patternedin the metal layers.

Making the sensor and optics small provides for mass-production of theimaging elements.

In some embodiments, the image sensors manufactured as described hereincan comprise monochromatic sensors. A monochromatic sensor can beconfigured to be sensitive to a targeted bandwidth of light, such as redlight, blue light, green light, yellow light, and or white light (or apanchromatic sensor). The monochromatic sensors can be manufactured on awafer-scale using wafer-scale optics as described herein. Color filtersor dyes can be applied to any suitable stage and/or any suitable layer.For example, the lens elements can be coated with a filter or the lenselements can include a dye which substantially filters light outside ofa targeted portion of the electromagnetic spectrum.

In some embodiments, the monochromatic sensors do not include anyfilters and are configured to be used in conjunction with a lens platethat includes monochromatic filters. When the lens plate and the sensorsare used in conjunction, monochromatic sensors are effectively created.In some embodiments, the lens plate is removable and allows fordifferent color filter schemes to be used to achieve a desired result.In some embodiments, a wafer is configured to contain at least fourimage sensors and a lens plate is configured to transmit a particularwavelength band to an associated image sensor. For example, the lensplate can make a first sensor a red sensor, transmitting red light whilesubstantially blocking all other wavelengths, the lens plate can make asecond sensor a blue sensor, a third sensor a green sensor, and a fourthsensor a white sensor. Control of the individual sensors can be arrangedthrough control electronics. In some embodiments, by making the imagesensors monochromatic, cross-talk is reduced and image qualityincreases, as described herein.

In some embodiments, any of the imaging systems described herein,including the dense field imagers, can include a camera array that isfabricated on a semiconductor chip. For example, U.S. Patent Publication2011/0080487 entitled “Capturing and Processing of Images UsingMonolithic Camera Array with Heterogeneous Imagers,” the entirety ofwhich is incorporated by reference herein and is included in theattached Appendix, describes the use of an array of imagers where eachimager in the array comprises a plurality of light sensing elements anda lens stack fabricated on a semiconductor chip. The camera array mayinclude two or more types of heterogeneous imagers, each imagerincluding two or more sensor elements or pixels. Each one of the imagersmay have different imaging characteristics. Alternatively, there may betwo or more different types of imagers where the same type of imagershares the same imaging characteristics.

In some embodiments, each imager has its own filter and/or opticalelement (e.g., lens). Specifically, each of the imagers or a group ofimagers may be associated with spectral color filters to receive certainwavelengths of light. Example filters include a traditional filter usedin the Bayer pattern (R, G, B or their complements C, M, Y), an IR-cutfilter, a near-IR filter, a polarizing filter, and a custom filter tosuit the needs of hyper-spectral imaging. Some imagers may have nofilter to allow reception of both the entire visible spectra andnear-IR, which increases the imager's signal-to-noise ratio. The numberof distinct filters may be as large as the number of imagers in thecamera array. Further, each of the imagers or a group of imagers mayreceive light through lenses having different optical characteristics(e.g., focal lengths) or apertures of different sizes.

In some embodiments, the imagers in the camera array are spatiallyseparated from each other by a separation distance. By increasing thespatial separation, the parallax between the images captured by theimagers may be increased. The increased parallax is advantageous wheremore accurate distance information is important. Separation between twoimagers may also be increased to approximate the separation of a pair ofhuman eyes. By approximating the separation of human eyes, a realisticstereoscopic 3D image may be provided to present the resulting image onan appropriate 3D display device.

In some embodiments, multiple camera arrays are provided at differentlocations on a device to overcome space constraints. One camera arraymay be designed to fit within a restricted space while another cameraarray may be placed in another restricted space of the device. Forexample, if a total of 20 imagers are required but the available spaceallows only a camera array of 1×10 imagers to be provided on either sideof a device, two camera arrays each including 10 imagers may be placedon available space at both sides of the device. Each camera array may befabricated on a substrate and be secured to a motherboard or other partsof a device. In addition, such imagers do not have to be homogenous insize, and may have different x- and y-dimensions. The images collectedfrom multiple camera arrays may be processed to generate images ofdesired resolution and performance.

In some embodiments, the imaging system can include wafer level optics.The wafer level optics can include a plurality of lens elements, whereeach lens element covers one of the sensors in the array. For example,the imaging system can be an array of pixels overlaid with color filtersand microlenses. The microlenses that sit on top of the color filtersare used to focus light on the active area of each underlying pixel. Themicrolenses can be thought of as sampling the continuous light field inobject space sampled by the main lens. Whereas the main lens samples thescene radiance light field, the micro-lenses sample the sensorirradiance light field.

The main lens associated with each imager maps the points in the objectspace to points in the image space such at that the mapping is bijective(onto-to-one and onto). Each microlens samples a finite extent of thesensor irradiance light field. The sensor irradiance light field iscontinuous and is the result of a bijective mapping from the objectspace. Thus, the microlens sampling of a finite extent of the sensorirradiance light field is also a sampling of a corresponding finiteextent of the scene radiance light field in object space.

In some embodiments, the imaging systems and dense field imagersdescribed herein can include monochromatic imaging elements. Themonochromatic imaging elements can include color filters that are offthe sensor, e.g., on the lens. The color filters can be modular, so thatdifferent filter plates can be switched and used with the same sensorarray. The filter plates can be added at the point of manufacture, or bythe consumer. Thus, the imaging system can include a plurality ofimaging elements that are assembled in a uniform fashion, e.g., sensorsthat are sensitive to a broad spectrum of light, and the imaging systemcan be customized by implementing a combination of filter arrays. Thistype of configuration can reduce manufacturing costs and/or increaseutility of the imaging system.

Example Embodiments

The following is a numbered list of example embodiments that are withinthe scope of this disclosure. The example embodiments that are listedshould in no way be interpreted as limiting the scope of theembodiments. Various features of the example embodiments that are listedcan be removed, added, or combined to form additional embodiments, whichare part of this disclosure:

Imaging Block and Dense Field Imager

1. An imaging block, comprising:

-   -   a support; at least a first imaging element and a second imaging        element carried by the support, each imaging element comprising        a sensor and a lens;    -   a mechanical connector for mechanically connecting the imaging        block into an array of imaging blocks; and    -   an electrical connector for electrically connecting the imaging        block into an array of imaging blocks.

2. An imaging block as in embodiment 1, wherein at least the firstimaging element comprises a monochromatic filter, wherein the sensor ofthe first imaging element detects substantially monochromatic lightpassing through the monochromatic filter.

3. An imaging block as in embodiment 1, wherein the support comprises awafer substrate.

4. An imaging block as in embodiment 3, wherein the wafer substratecomprises a semi conductor material.

5. An imaging block as in embodiment 4, wherein the sensors are formedon the substrate.

6. An imaging block as in embodiment 3, wherein the lenses arewafer-level lenses.

7. An imaging array, comprising an array support, and at least twoimaging blocks of any of embodiments 1 through 6 carried by the arraysupport, the at least two imaging blocks comprising a first imagingblock and a second imaging block.

8. An imaging array as in embodiment 7, wherein at least one of thesensors in the first imaging block is non-coplanar with at least one ofthe sensors in the second imaging block.

9. An imaging array as in embodiment 7, wherein each of the sensors inthe first imaging block are coplanar, each of the sensors in the secondimaging block are coplanar, and each of the sensors in the first imagingblock are non-coplanar with each of the sensors in the second imagingblock.

10. An imaging array as in embodiment 7, wherein each imaging elementhas a primary optical axis, and the primary optical axis of at least oneof the imaging elements in the first imaging block is substantiallynon-parallel with the primary optical axis of at least one of theimaging elements in the second imaging block.

11. An imaging array as in embodiment 7, wherein each imaging elementhas a primary optical axis, the primary optical axes of the imagingelements in the first imaging block are substantially parallel, theprimary optical axes of the imaging elements in the second imaging blockare substantially parallel, and the primary optical axes of the imagingelements in the first imaging block are substantially non-parallel withthe primary optical axes of the imaging elements in the second imagingblock.

12. An imaging array as in embodiment 7, wherein a primary optical axisof at least one imaging element of the first imaging block is angularlyadjustable with respect to a primary optical axis of at least oneimaging element of the second imaging block.

13. An imaging array as in embodiment 12, wherein the imaging arraycomprises a user-actuatable control for achieving the angularadjustment.

14. An imaging array as in embodiment 7, further comprising:

-   -   an image processing system configured to:        -   using image data captured by the first imaging block,            generate a first image data set representative of a first            portion of a light field; and        -   using image data captured by second imaging block, generate            a second image data set representative of a second portion            of the light field; and        -   derive a third image data set from the first and second            image data sets.

15. An imaging array as in embodiment 14, wherein the image processingsystem is carried by the array support.

16. An imaging array as in embodiment 14, wherein the image processingsystem is physically separate from the array support and receives thefirst and second image data sets wirelessly.

17. An imaging array as in embodiment 14, wherein the image processingsystem derives the third image data set at least partly by creating aspatial relationship tensor that includes spatial relationshipinformation between elements of the first and second image data sets andusing the spatial relationship tensor to derive the third image dataset.

18. An imaging array as in embodiment 17, wherein the image processingsystem derives the third image data set at least partly by using thespatial relationship tensor to combine together the first and secondportions of the light field.

19. An imaging block as in embodiment 1, further comprising one or moreprocessors carried by the support and configured to generate a lightfield representation based on pixel data acquired from the sensors.

20. An imaging block as in embodiment 1, wherein one or more of thelenses are removably replaceable with lenses having different opticalcharacteristics.

21. An imaging block as in embodiment 1, wherein each imaging elementhas a primary optical axis, and most of the primary optical axes aresubstantially parallel.

22. An imaging block as in embodiment 1, wherein each imaging elementhas a primary optical axis, and at least two of the primary optical axesdiverge in a direction leading away from the sensor.

23. An imaging block as in embodiment 1, wherein each imaging elementhas a primary optical axis, and at least two of the primary optical axesconverge in a direction leading away from the sensor.

24. An imaging block as in embodiment 1, wherein each imaging elementhas a primary optical axis, and at least a first primary optical axis isangularly adjustable with respect to at least a second primary opticalaxis.

25. An imaging block as in embodiment 1, comprising at least 8 imagingelements.

26. An imaging block as in embodiment 25, wherein the imaging elementsare arranged in two rows of 4.

27. An imaging block as in embodiment 1, comprising at least 16 imagingelements.

28. An imaging block as in embodiment 27, wherein the imaging elementsare arranged in a 4×4 grid.

29. An imaging block as in embodiment 1, wherein at least one sensor isno larger than about 5 mm×5 mm.

30. An imaging block as in embodiment 1, further comprising an FPGA chipcarried by the support.

31. An imaging array as in embodiment 1, wherein at least some of thesensors have one or more of different sizes, different resolutions, ordifferent sensitivities.

32. An imaging array comprising an array support, and at least twoimaging blocks of any of claims 19 through 31 carried by the arraysupport.

Method of Compiling an Image Data Set

1. A method of compiling an image data set, comprising:

-   -   obtaining a first image data set representative of a first        portion of a light field;    -   obtaining a second image data set representative of a second        portion of the light field;    -   with one or more processors, deriving a third image data set        that is based on at least the first image data set, the second        image data set, and information relating to a spatial        relationship between the first and second portions of the light        fields; and storing the third image data set in one or more        memory devices.

2. A method as in embodiment 1, wherein the first image data set isderived from pixel data acquired by a first group of at least twoimaging elements and the second image data set is derived from pixeldata acquired by a second group of at least two imaging elements, eachimaging element comprising a sensor and a lens.

3. A method as in embodiment 1, further comprising processing the firstimage data set and the second image data set to determine the spatialrelationship between the first portion of the light field and the secondportion of the light field.

4. A method as in embodiment 1, wherein said deriving comprisesaccessing a spatial relationship tensor representing the spatialrelationship information to perform a geometric transform on the firstimage data set and second image data set.

5. A method as in embodiment 2, wherein the at least two imagingelements in the first group are coplanar, the at least two imagingelements in the second group are coplanar, and the at least two imagingelements in the first group are non-coplanar with respect to the atleast two imaging elements in the second group.

6. A method as in embodiment 1, wherein the first and second image datasets additionally represent the first portion and the second portion ofthe light field as a function of time.

7. A method as in embodiment 1, wherein the third image data setcomprises light field information represented as a function of time.

8. A method as in embodiment 1, wherein the first portion of the lightfield and the second portion of the light field comprise regions of thelight field which at least partially overlap, and wherein the third dataset comprises light field information derived from data in both of thefirst and second data sets that corresponds to the region of the lightfield lying within the overlap.

9. A method as in embodiment 1, wherein the first portion of the lightfield and the second portion of the light field comprise regions of thelight field which only partially overlap, and wherein the third data setcomprises light field information derived from:

-   -   data in the first image data set that corresponds to a first        portion of a scene; and    -   data in the second image data set that corresponds to a second        portion of the scene that does not overlap with the first        portion.

10. A method as in embodiment 1, wherein the third image data setcomprises at least 4D light field information.

11. A method as in embodiment 1, wherein said deriving the third imagedata set comprises deriving the third image data set while maintainingthe dimensionality of functions that represent the first and secondportions of the light field.

12. A method as in embodiment 1, wherein the first and second image datasets respectively represent the first and second portions of the lightfield as functions having at least four input parameters.

13. A method as in embodiment 1, wherein the first and second image datasets respectively represent the first and second portions of the lightfield as functions having at least five input parameters.

14. A method as in embodiment 1, wherein the first and second image datasets respectively represent the first and second portions of the lightfield as functions that represent luminance as a function of a positionin space and a pointing direction.

15. A method of embodiment 1, wherein viewable images are extractablefrom the third image data set.

16. A method as in embodiment 15, wherein the viewable images comprise2D images.

17. A method as in embodiment 15, wherein the viewable images comprise3D images.

18. A method as in embodiment 1, wherein viewable motion video isextractable from the third image data set.

19. A method as in embodiment 1, further comprising accessing a fourthimage data set representative of a third portion of the light field,wherein said deriving comprises deriving the third image data set basedon at least the first image data set, the second image data set, thefourth image data set, and spatial relationships between the first,second, and third portions of the light field.

Stored Image Data Set

1. A memory device containing a dense field image data set, comprising:

-   -   a storage medium; and    -   a dense field image data set stored in the storage medium and        derived by relating a first image data set representative of a        first portion of a light field to a second image data set        representative of a second portion of the light field, using        information relating to a spatial relationship between the first        and second light fields.

2. A computer-readable memory device as in embodiment 1, wherein theimage data set comprises light field information represented as afunction of time.

3. A computer-readable memory device as in embodiment 1, wherein theimage data set comprises at least 4D light field information.

4. A computer-readable memory device as in embodiment 1, wherein thefirst image data set was derived from pixel data acquired by a firstimaging block comprising a support and at least two imaging elementscarried by the support, and the second image data set was derived frompixel data acquired by a second imaging block comprising a support andat least two imaging elements carried by the support, each imagingelement comprising a sensor and a lens.

5. A machine comprising the computer-readable memory device ofembodiment 1 and one or more processors, the one or more processorsconfigured to derive the dense field image data set.

6. A machine comprising the computer-readable memory device ofembodiment 1 and one or more processors, the one or more processorsconfigured to extract viewable images from the dense field image dataset.

7. A computer-readable memory device as in embodiment 6, wherein theviewable images comprise 2D images.

8. A computer-readable memory device as in embodiment 6, wherein theviewable images comprise 3D images.

9. A computer-readable memory device as in embodiment 6, wherein theviewable images comprise motion video.

Stitching Light Fields to Create Dense Field Image Set with an EnhancedField of View

1. A method of creating a dense field image set, the method comprising:

-   -   acquiring pixel data from a plurality of imaging elements, each        imaging element comprising a sensor and a lens;    -   generating a plurality of light field representations, each of        the light field representations generated using pixel data from        at least two of the imaging elements;    -   creating a spatial relationship tensor representative of spatial        relationships among the light field representations; and    -   utilizing the spatial relationship tensor, combining the light        field representations to create a dense field image set.

2. A method as in embodiment 1, wherein said combining the light fieldrepresentations comprises stitching together light field representationshaving at least partially non-overlapping fields of view to create adense field image set having a significantly wider field of view thanthe individual light field representations.

3. A method as in embodiment 2, wherein the lateral field of view of thedense field image set is greater than or equal to about 145 degrees.

4. A method as in embodiment 1, wherein the imaging elements arearranged on a common support.

5. A method as in embodiment 1, wherein at least some of the imagingelements are not coplanar.

6. A method as in embodiment 1, wherein each light field representationis generated using pixel data from an imaging block comprising at leasttwo imaging elements.

7. A method as in embodiment 6, wherein at least some of the imagingblocks are not coplanar.

8. A method as in embodiment 1, wherein at least one of the imagingelements comprises a monochromatic filter, and wherein light passingthrough the monochromatic filter is detected by the sensor of the atleast one imaging element.

9. A method as in embodiment 1, wherein at least half of the imagingelements comprise monochromatic filters.

10. A method as in embodiment 1, wherein substantially all of theimaging elements comprise monochromatic filters.

Dense Field Imager Creating Light Fields from Multiple Tiles Using PixelCorrespondence Information

1. An imaging system, comprising:

-   -   a plurality of imaging blocks each comprising at least two        imaging elements, each of the imaging elements comprising an        image sensor and a lens;    -   a dense field image processor module configured to:        -   for each imaging block of at least two of the plurality of            imaging blocks:            -   generate pixel correspondence information for the                imaging block, the pixel correspondence information                representative of spatial relationships between pixels                in each imaging element of the imaging block and                corresponding pixels in other imaging elements of the                imaging block;            -   utilizing the correspondence information, generate a                light field representation using pixel data acquired by                the respective imaging block.

2. An imaging system as in embodiment 1, wherein the dense field imageprocessor module is further configured to:

-   -   create a spatial relationship tensor representative of spatial        relationships among the light field representations; and    -   utilizing the spatial relationship tensor, combine the light        field representations to create a dense field image set.

3. An imaging system as in embodiment 1, wherein the imaging blocks arearranged on a common support.

4. An imaging system as in embodiment 1, wherein the imaging blocks areformed in a wafer.

5. An imaging system as in embodiment 2, wherein the imaging elements ofeach imaging block are coplanar with respect to one another.

6. An imaging system as in embodiment 2, wherein at least some of theimaging blocks are not coplanar with respect to other ones of theimaging blocks.

7. An imaging system as in embodiment 1, wherein at least some of theimaging blocks are arranged on physically separate supports.

Dense Filed Imager Capable of Stitching Light Fields

1. A dense field imaging system, comprising:

-   -   a plurality of imaging blocks, each of the imaging blocks        comprising at least two sensor/lens pairs, wherein at least some        of the imaging blocks are substantially non-coplanar with each        other and have at least partially non-overlapping fields of        view;    -   a dense field image processor module configured to:        -   for each imaging block, generate a light field            representation using pixel data acquired by the sensor/lens            pairs;        -   generate a spatial relationship tensor representative of            spatial relationships between the light field            representations; and using the spatial relationship tensor,            combine the light field representations to create a dense            field imaging set having a substantially wider field of view            than the individual light field representations.

2. A dense field imager as in embodiment 1, wherein each of the imagingblocks comprises a wafer substrate on which the sensors are formed.

3. A dense field imager as in embodiment 1, further comprising a controlconfigured to adjust the angular relationship between sensor/lens pairs.

Dense Field Imaging System with Monochromatic Sensors

1. A dense field imaging system, the dense field imaging systemcomprising:

-   -   a plurality of imaging blocks arranged on a support, each of the        imaging blocks comprising at least two imaging elements, each of        the imaging elements comprising a sensor and lens, wherein at        least some of the imaging elements comprise monochromatic        filters; and    -   a dense field image processor module configured to:        -   for each imaging block, generate a light field            representation using pixel data acquired by the imaging            elements of the imaging block; and        -   combine the light field representations to create a dense            field image set.

2. A dense field imaging system as in embodiment 1, wherein most of thesensors are monochromatic.

3. A dense field imaging system as in embodiment 1, wherein each of thesensors is monochromatic.

4. A dense field imaging system as in embodiment 1, wherein each of theimaging blocks includes a support comprising a wafer substrate, andwherein the sensors are formed on the wafer substrate.

5. A dense field imaging system as in embodiment 4, wherein the lensescomprise wafer level lenses.

Terminology/Additional Embodiments

Embodiments have been described in connection with the accompanyingdrawings. However, it should be understood that the figures are notdrawn to scale. Distances, angles, etc. are merely illustrative and donot necessarily bear an exact relationship to actual dimensions andlayout of the devices illustrated. In addition, the foregoingembodiments have been described at a level of detail to allow one ofordinary skill in the art to make and use the devices, systems, etc.described herein. A wide variety of variation is possible. Components,elements, and/or steps can be altered, added, removed, or rearranged.While certain embodiments have been explicitly described, otherembodiments will become apparent to those of ordinary skill in the artbased on this disclosure.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orstates. Thus, such conditional language is not generally intended toimply that features, elements and/or states are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or states are included or are to beperformed in any particular embodiment.

Depending on the embodiment, certain acts, events, or functions of anyof the methods described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not alldescribed acts or events are necessary for the practice of the method).Moreover, in certain embodiments, acts or events can be performedconcurrently, e.g., through multi-threaded processing, interruptprocessing, or multiple processors or processor cores, rather thansequentially. In some embodiments, the algorithms disclosed herein canbe implemented as routines stored in a memory device. Additionally, aprocessor can be configured to execute the routines. In someembodiments, custom circuitry may be used.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. The described functionalitycan be implemented in varying ways for each particular application, butsuch implementation decisions should not be interpreted as causing adeparture from the scope of the disclosure.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein can be implementedor performed with a general purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general purpose processor can be a microprocessor, but in thealternative, the processor can be any conventional processor,controller, microcontroller, or state machine. A processor can also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

The blocks of the methods and algorithms described in connection withthe embodiments disclosed herein can be embodied directly in hardware,in a software module executed by a processor, or in a combination of thetwo. A software module can reside in RAM memory, flash memory, ROMmemory, EPROM memory, EEPROM memory, registers, a hard disk, a removabledisk, a CD-ROM, or any other form of computer-readable storage mediumknown in the art. An exemplary storage medium is coupled to a processorsuch that the processor can read information from, and write informationto, the storage medium. In the alternative, the storage medium can beintegral to the processor. The processor and the storage medium canreside in an ASIC. The ASIC can reside in a user terminal. In thealternative, the processor and the storage medium can reside as discretecomponents in a user terminal.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it will beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As will berecognized, certain embodiments of the inventions described herein canbe embodied within a form that does not provide all of the features andbenefits set forth herein, as some features can be used or practicedseparately from others. The scope of certain inventions disclosed hereinis indicated by the appended claims rather than by the foregoingdescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

What is claimed is:
 1. An imaging array, comprising: first and secondimaging blocks each comprising: a support; and at least three imagingelements carried by the support, each imaging element of the at leastthree imaging elements comprising a lens and an image sensor comprisinga plurality of pixels; wherein the first and second imaging blocks aremounted with respect to one another such that one or more of the supportof the first imaging block and the support of second imaging block aremovable to adjust an orientation of the first and second imaging blocksrelative to one another; and wherein the image sensor of at least one ofthe at least three imaging elements of the first imaging block isnon-coplanar with the image sensor of at least one of the at least threeimaging elements of the second imaging block.
 2. An imaging array as inclaim 1, wherein at least one of the imaging elements of the firstimaging block comprises a monochromatic filter and detects substantiallymonochromatic light passing through the monochromatic filter.
 3. Animaging array as in claim 1, wherein the support of the first imagingblock comprises a wafer substrate and the support of the second imagingblock comprises a wafer substrate.
 4. An imaging array as in claim 3,wherein the wafer substrate of the support of the first imaging blockcomprises a semiconductor material and the wafer substrate of thesupport of the second imaging block comprises a semiconductor material.5. An imaging array as in claim 4, wherein the image sensors of the atleast three imaging elements of the first imaging block are formed onthe wafer substrate of the support of the first imaging block and theimage sensors of the at least three imaging elements of the secondimaging block are formed on the wafer substrate of the support of thesecond imaging block.
 6. An imaging array as in claim 3, wherein thelenses of the at least three imaging elements of the first imaging blockare wafer-level lenses and the lenses of the at least three imagingelements of the second imaging block are wafer-level lenses.
 7. Animaging array as in claim 1, wherein each imaging element of the atleast three imaging elements of the first and second imaging blocks hasa primary optical axis, and the primary optical axis of at least one ofthe imaging elements in the first imaging block is substantiallynon-parallel with the primary optical axis of at least one of theimaging elements in the second imaging block.
 8. An imaging array as inclaim 1, wherein each imaging element of the at least three imagingelements of the first and second imaging blocks has a primary opticalaxis, the primary optical axes of the imaging elements in the firstimaging block are substantially parallel with one another, the primaryoptical axes of the imaging elements in the second imaging block aresubstantially parallel with one another, and the primary optical axes ofthe imaging elements in the first imaging block are substantiallynon-parallel with the primary optical axes of the imaging elements inthe second imaging block.
 9. An imaging array as in claim 1, wherein aprimary optical axis of at least one imaging element of the firstimaging block is angularly adjustable with respect to a primary opticalaxis of at least one imaging element of the second imaging block.
 10. Animaging array as in claim 1, further comprising an image processingsystem configured to: receive first image data image data generated bythe first imaging block; receive second image data generated by thesecond imaging block; and derive third image data from the first andsecond image data.
 11. An imaging array as in claim 10, wherein thefirst imaging block comprises one or more processors carried by thesupport of the first imaging block configured to generate the firstimage data, and the second imaging block comprises one or moreprocessors carried by the support of the second imaging block configuredto generate the second image data.
 12. An imaging array as in claim 1,further comprising one or more processors configured to generate a lightfield representation based on pixel data acquired from the image sensorsof the first and second imaging blocks.
 13. An imaging array as in claim1, wherein one or more of the lenses of the at least three imagingelements of the first imaging block are removably replaceable withlenses having different optical characteristics.
 14. An imaging array asin claim 1, wherein at least one of the at least three imaging elementsof the first imaging block comprises a fixed optical power and at leastanother imaging element of the at least three imaging elements of thefirst imaging block comprises a variable optical power.
 15. An imagingarray as in claim 1, wherein at least one of the at least three imagingelements of the first imaging block is configured to capture light of afirst wavelength and at least another imaging element of the at leastthree imaging elements of the first imaging block is configured tocapture light of a different wavelength than the first wavelength. 16.An imaging block as in claim 1, wherein the first and second imagingblocks each comprise at least 16 imaging elements.
 17. An imaging arrayas in claim 1, wherein the at least three imaging elements of each ofthe first imaging block and the second imaging block are arranged in atleast one of a rectangular grid, a hexagonal grid, and concentriccircles.